Method and apparatus for acquisition, compression, and characterization of spatiotemporal signals

ABSTRACT

The present invention provides methods and apparatus for acquisition, compression, and characterization of spatiotemporal signals. In one aspect, the invention assesses self-similarity over the entire length of a spatiotemporal signal, as well as on a moving attention window, to provide cost effective measurement and quantification of dynamic processes. The invention also provides methods and apparatus for measuring self-similarity in spatiotemporal signals to characterize, adaptively control acquisition and/or storage, and assign meta-data for further detail processing. In some embodiments, the invention provides for an apparatus adapted for the characterization of biological units, and methods by which attributes of the biological units can be monitored in response to the addition or removal of manipulations, e.g., treatments. The attributes of biological units can be used to characterize the effects of the abovementioned manipulations or treatments as well as to identify genes or proteins responsible for, or contributing to, these effects.

CLAIM OF PRIORITY

[0001] This application claims priority under 35 USC §119(e) to U.S.Patent Application Serial No. 60/356,317, filed on Feb. 13, 2002, theentire contents of which are hereby incorporated by reference.

FIELD OF THE INVENTION

[0002] The present invention relates to methods and apparatus forcharacterizing dynamic systems.

BACKGROUND OF THE INVENTION

[0003] Images over time, also known as video, capture our daily lives,industrial processes, environmental conditions, etc, economically andaccessibly. Compression systems can significantly reduce the cost oftransmitting lengthy videos. Machine vision systems can register imageswith accuracy of fractions of a pixel. Supervised cataloging systems canorganize and annotate hours and hours of video for efficient re-use.

[0004] Many scientific and industrial applications would benefit fromexploiting cost effective video systems for better measurement andquantification of dynamic processes. The current techniques require highcomputational and storage costs and do not allow for a real-timeassessment and control of many nonlinear dynamic systems.

[0005] The present invention relates generally to digital data andsignal processing. It relates more particularly, by way of example, tomeasuring self similarity in spatiotemporal signals to characterize(cluster, classify, represent), adaptively control their acquisitionand/or storage and assign meta-data and further detail processing. Italso relates to qualitative and/or quantitative assessment ofspatiotemporal sensory measurements of dynamic systems.

SUMMARY OF THE INVENTION

[0006] In general, the invention features methods, e.g., machine-basedmethods, and apparatuses for evaluating a dynamic system. The methodscan include one or more of the following steps (the steps need not bebut typically are performed in the order provided herein): acquiring aplurality of images representative of the dynamic system in two or moredimensions, e.g., three dimensions; determining similarity between aselected image and one of the other images; and characterizing theselected image as a statistical function of the similarity determinedwith respect to it, thereby characterizing the dynamic system, e.g.,characterizing the selected image as a function of similarity to one ormore images acquired from a different part of the two dimensionalcontinuum, e.g., one or both of an earlier acquired image and/or a lateracquired image. In the present methods, a selected image can be comparedwith one or a plurality of other images, e.g., N images, wherein N isselected by the user and can be any number between 1 and the totalnumber of images acquired, e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15,16, 18, 20, 30, 40, 50, 60, 70, 80, 90, 100 or more, and any number inbetween. The two dimensions can include any dimensions, including butnot limited to time, frequency spectrum, temperature, presence orabsence of an attribute of the system. The determining step can includedetermining similarity between each image and each of the other images;and the characterizing step can include characterizing the dynamicsystem as a statistical function of the similarities determined withrespect to the plurality of images.

[0007] Although many of the embodiments described herein refer to timeas the two dimensional system it should be understood that analogousembodiments, which acquire images in other dimensions, are included inthe invention.

[0008] In some embodiments of the invention, the images are acquired byan attentive acquisition or storage method including some or all of thefollowing (the steps need not be but typically are performed in theorder provided herein): acquiring images at an initial acquisitionand/or storage parameterization, e.g., a first or selectedparameterization; determining similarity between selected images, e.g.,a more recently acquired image and at least one of the other images,e.g., one or more previously acquired images, e.g., N previouslyacquired images, where N is set by the user, and can be any numberbetween one and all of the previously acquired images; characterizingthe selected images as a statistical function of self-similarity;optionally comparing the characterization with a reference value, e.g.,a pre-selected reference value, and optionally adjusting the acquisitionor storage parameterization as a function of the self-similarity of theimages. In some embodiments, the pre-selected reference value is ameasure of change and/or the rate of change in the dynamic system, e.g.,self-similarity.

[0009] In another aspect, the present invention features methods, e.g.,machine-based methods, for evaluating a dynamic system over time. Themethod includes one or more of and preferably all of (the steps need notbe but typically are performed in the order provided herein): acquiringa plurality of images representative of the dynamic system over time;determining similarity between a selected image and one of the otherimages; and characterizing the selected image as a statistical functionof the similarity determined with respect to it. The determining stepcan include determining similarity between each image and each of theother images; and the characterizing step can include characterizing thedynamic system as a statistical function of the similarities determinedwith respect to the plurality of images.

[0010] In another aspect, the present invention provides methods, e.g.,machine-based methods, for evaluating a dynamic system over timeincludes some or all of (the steps need not be but typically areperformed in the order provided herein): acquiring a plurality of imagesrepresentative of the dynamic system in two or more, e.g., threedimensions, such as time, space, or time and space; determiningself-similarity among a representative set of images; and characterizingthe set of images as a statistical function of self-similarity. The twodimensions can include any of time, space, frequency spectrum,temperature, presence or absence of an attribute of the system. In someembodiments, the determining step can include determiningself-similarity between some or all of the plurality of images; and thecharacterizing step can include characterizing the dynamic system as astatistical function of the self-similarities determined with respect tothe plurality of images. In some embodiments, the images are acquired bya method comprising acquiring images at an initial acquisition and/orstorage parameterization; determining similarity between selectedimages; characterizing the selected images as a statistical function ofself-similarity; optionally comparing the characterization with areference value; and optionally adjusting the acquisition or storageparameterization as a function of the self-similarity of the images. Insome embodiments, the pre-selected reference value is a measure ofchange and/or the rate of change in the dynamic system, e.g.,self-similarity.

[0011] In another aspect, the present invention features methods, e.g.,machine-based methods, for evaluating a dynamic system. The methodincludes one or all, typically all, of the following (the steps need notbe but typically are performed in the order provided herein): acquiringa plurality of images representative of the dynamic system over time;determining self-similarity among a representative set of images, e.g.,some or all of the images, e.g., every other image, every third image,randomly selected images, etc.; and characterizing the set of images asa statistical function of self-similarity.

[0012] In some embodiments, the determining step can include determiningself-similarity between all of the plurality of images; and thecharacterizing step can include characterizing the dynamic system as astatistical function of the self-similarities determined with respect tothe plurality of images.

[0013] In some embodiments, the images are acquired by a method, e.g., amachine-based method, comprising acquiring images at an initialacquisition and/or storage parameterization; determining similaritybetween selected images; characterizing the selected images as astatistical function of self-similarity; optionally comparing thecharacterization with a reference value; and optionally adjusting theacquisition or storage parameterization as a function of theself-similarity of the images. In some embodiments, the pre-selectedreference value is a measure of change and/or the rate of change in thedynamic system, e.g., self-similarity. In some embodiments, thestatistical function is a measure of entropy. In some embodiments, thestatistical function is Shannon's entropy function. In some embodiments,the statistical function is:

H _(j) =−ΣP _(j) log ₂(P _(j))/log2(n)  (10).

[0014] In some embodiments, the determining step can include determiningpair-wise correlations between images, e.g., pairs of images, forexample, determining pair-wise correlations between a plurality ofimages that comprise a window of length n images. In some embodiments,the determining step includes approximating a correlation between imagesseparated by more than n by treatment of intervening pair-wisecorrelations as transitional probabilities. In some embodiments, thedetermining step can include determining long-term and short-termpair-wise correlations between images. In some embodiments, thedetermining step can include generating a matrix of the similarities.The determining step can include generating a matrix, e.g., acorrelation matrix, that is any of square, normalized, comprised ofprobabilities, and has a diagonal of ones. In further embodiments, themethod includes applying a matrix operation to the matrix in order tocharacterize the dynamic system.

[0015] In some embodiments of the invention, the images can be acquiredfrom a sensor. The sensor can be any sensor known in the art, includingbut not limited to a video camera or other device suitable foracquisition of spatiotemporal or other signals, regardless of whetherthose signals represent the visual spectrum. The images can be acquiredby any method known in the art, and can include any of (i) an imagecaptured by a sensor, and (ii) a processed form of an image captured bya sensor. The processed form of the image can be any processed imageknown in the art, including but not limited to (i) a filtered form of animage captured by the sensor, (ii) a windowed form of the image capturedby the sensor, (iii) a sub-sampled form of the image, (iv) anintegration of images captured by the sensor over time, (v) anintegration of a square of images captured by the sensor over time, (vi)a gradient-direction form of the image, and/or (vii) a combinationthereof.

[0016] In another aspect, the invention features a method, e.g., amachine-based method, of attentively acquiring or storing imagesrepresentative of a dynamic system over time. The method includes someor all, typically all, of the following steps (the steps need not be buttypically are performed in the order provided herein): acquiring, at aselected acquisition and/or storage parameterization, a plurality ofimages representative of the dynamic system over time; determiningsimilarity between a selected image and at least one of the otherimages; characterizing the images as a statistical function ofself-similarity; optionally comparing the characterization with areference value, e.g., a pre-selected reference value, and optionallyadjusting the acquisition and/or storage parameterization as a functionof the self-similarity of the images. In some embodiments, theacquisition parameterization can be set to drive the statisticalfunction to a predetermined level, e.g., close to zero. In someembodiments, the acquisition parameterization can be set so that atleast one or more most recently acquired images reflects a predeterminedrate of change. In some embodiments, the acquisition parameterizationcan be set so that at least one or more most recently acquired imagesreflects a predetermined rate of motion, shape change, focal change,temperature change, intensity change.

[0017] Thus in some embodiments of the invention, the images areacquired by an attentive acquisition or storage method including some orall of the following (the steps need not be but typically are performedin the order provided herein): acquiring images at a first acquisitionparameterization; determining similarity between a selected image, e.g.,a more recently acquired image, and at least one of the other images,e.g., one or more previously acquired images, e.g., N previouslyacquired images, where N is set by the user, and can be any numberbetween one and all of the previously acquired images; characterizingthe images as a statistical function of self-similarity; optionallycomparing the characterization with a reference value, e.g., apre-selected reference value, and optionally adjusting the acquisitionor storage parameterization as a function of the self-similarity of theimages. In some embodiments, the pre-selected reference value is ameasure of change and/or the rate of change in the dynamic system, e.g.,self-similarity.

[0018] In some embodiments, the acquisition parameterization includes,but is not limited to, any of acquisition rate, exposure, aperture,focus, binning, or other parameter.

[0019] In some embodiments, at least selected ones of the acquiredimages are buffered for potential processing. In some embodiments, atleast selected ones of the buffered images are processed. In someembodiments, at least selected ones of the acquired images are stored.

[0020] In another aspect, the present invention features a method, e.g.,a machine-based method, of determining movement of an object. The methodincludes some or all of the following (the steps need not be buttypically are performed in the order provided herein): acquiring aplurality of images of the object; selecting a window of interest in aselected image, the selecting step including performing at least oneautocorrelation between a candidate window and a region in which thecandidate window resides in the selected image; identifying movement ofthe object as function of a cross-correlation between the window ofinterest and corresponding window in another of the images, e.g., byperforming at least one autocorrelation between a candidatecorresponding window in the another image and a region in that image inwhich that candidate window resides, optionally by finding a maxima inthe cross-correlation. The images can be acquired by a method describedherein, including a method including attentive acquisition or storage,wherein the storage or acquisition parameterizations are optionallyadjusted as a function of the self-similarity of some subset of theacquired images.

[0021] In another aspect, the present invention provides a method, e.g.,a machine-based method, for determining movement of an object. Themethod includes some or all of the following (the steps need not be buttypically are performed in the order provided herein): acquiring aplurality of images of the object; selecting a window of interest in aselected image, the selecting step including performing at least oneautocorrelation between a candidate window and a region in which thecandidate window resides in the selected image; performing at least oneautocorrelation on a window that corresponds to the window of interestin another of the images; and identifying movement of the object asfunction of displacement of the characterizing portions of theautocorrelations, e.g., by matching at least characterizing portions ofthe autocorrelations. In some embodiments, the images are acquired by amethod including attentive acquisition or storage, wherein the storageor acquisition parameterization are optionally adjusted as a function ofthe self-similarity of some subset of the acquired images.

[0022] In another aspect, the present invention provides a method, e.g.,a machine-based method, of analyzing motion in a plurality of images.The method includes some or all of the following (the steps need not bebut typically are performed in the order provided herein): acquiring aplurality of images, selecting a plurality of windows of interest in aselected image, the selecting step including performing, for each windowof interest, at least one autocorrelation between a candidate window anda region in which the candidate window resides in the selected image;and identifying motion vectors as function of a cross-correlationbetween each window of interest and a corresponding window in another ofthe images, e.g., by performing at least one autocorrelation between acandidate corresponding window in another image and a region in thatimage in which that candidate window resides, and optionally finding amaxima in the cross-correlations. In some embodiments, the images areacquired by a method including attentive acquisition or storage, whereinthe storage or acquisition parameterizations are optionally adjusted asa function of the self-similarity of some subset of the acquired images.In some embodiments, the method also includes segmenting the image as afunction of the motion vectors, e.g., by finding one or more sets ofmotion vectors with minimum square distances with respect to oneanother.

[0023] In another aspect, the invention provides a method, e.g., amachine-based method of analyzing motion in a plurality of images. Themethod includes some or all, typically all, of the following (the stepsneed not be but typically are performed in the order provided herein):acquiring a plurality of images of the object; selecting a plurality ofwindows of interest in a selected image, by performing, for each windowof interest, at least one autocorrelation between a candidate window anda region in which the candidate window resides in the selected image;for each window of interest, performing at least one autocorrelation ona respective corresponding window in another image; and identifyingmotion vectors as functions of displacements of the characterizingportions the autocorrelations of each window of interest and thecorresponding window in the another image, e.g., by matching at leastcharacterizing portions of the autocorrelations. In some embodiments,the images are acquired by a method including acquiring images at aninitial acquisition and/or storage parameterization; determiningsimilarity between selected images; characterizing the selected imagesas a statistical function of self-similarity; optionally comparing thecharacterization with a reference value; and optionally adjusting theacquisition or storage parameterization as a function of theself-similarity of the images. In some embodiments, the pre-selectedreference value is a measure of change and/or the rate of change in thedynamic system, e.g., self-similarity.

[0024] In some embodiments, the method also includes segmenting theimage based on self-similarity, e.g., as a function of the motionvectors, e.g., by finding one or more sets of motion vectors withminimum square distances with respect to one another.

[0025] In the methods of the present invention, the dynamic system is adynamic biological system including at least one biological unit asdefined herein. In some embodiments, the biological unit is undergoingmorphological change, e.g., cell differentiation, spreading,contraction, phagocytosis, pinocytosis, exocytosis, growth, death,division, and polarization.

[0026] In some embodiments of the present invention, the dynamicbiological system is in a single well, e.g., one or more wells, e.g.,one or more wells of a dish having multiple wells. In some embodiments,the biological units are on an addressable array, e.g., a cell chip, amulti-well plate, e.g., 96 wells, etc.

[0027] In some embodiments of the present invention, the plurality ofimages representative of the dynamic system are images of a singlebiological unit.

[0028] In some embodiments, the biological unit is motile.

[0029] In some embodiments, the biological unit is undergoing celldivision, e.g., undergoing meiosis or mitosis.

[0030] In some embodiments, the biological unit is undergoing celladherence, e.g., is adjacent to, in contact with, or adhered to a secondentity during image acquisition. The second entity can be a surface oranother biological unit.

[0031] In some embodiments, the biological units are subcellular objectssuch as proteins, nucleic acids, lipids, carbohydrates, ions, ormulticomponent complexes containing any of the above. Further examplesof subcellular objects include organelles, e.g., mitochondria, Golgiapparatus, endoplasmic reticulum, chloroplast, endocytic vesicle,exocytic vesicles, vacuole, lysosome, nucleus. In some embodiments, thebiological unit is labeled, e.g., with magnetic or non-magnetic beads,antibodies, fluorophores, radioemitters, and labeled ligands. Theradioemitter can be an alpha emitter, a beta emitter, a gamma emitter,or a beta- and gamma-emitter. The label can be introduced into thebiological unit using any method known in the art, includingadministering to cells or ogranisms, by injecting, incubating,electroporating, soaking, etc. Labelled biological units can also bederived synthetically, chemically, enzymatically, or genetically, e.g.,by creation of a transgenic animal expressing GFP in one or more cells,or expressing a GFP-tagged protein in one or more cells. The label canalso be chemically attached, e.g., a labelled antibody or ligand.

[0032] In one aspect, the present invention provides a method, e.g., amachine-based method, for evaluating an attribute of a biological unitover time. The method includes, some or all, typically all, of thefollowing (the steps need not be but typically are performed in theorder provided herein): providing a plurality of images representativeof the biological unit over time; evaluating the similarity between aselected image and one of the other images to determine a pairwisesimilarity measurement, e.g., by computed pairwise correlations or byemploying fourier optics; generating a self-similarity matrix comprisingthe pairwise similarity measurement; and characterizing the biologicalunit as a function of the self-similarity matrix, e.g., by generatingeigenvalues and/or entropic indices from the self-similarity matrix,thereby evaluating the attribute of the biological system. In someembodiments, the images are acquired by a method acquiring images at aninitial acquisition and/or storage parameterization; determiningsimilarity between selected images; characterizing the selected imagesas a statistical function of self-similarity; optionally comparing thecharacterization with a reference value; and optionally adjusting theacquisition or storage parameterization as a function of theself-similarity of the images. In some embodiments, the pre-selectedreference value is a measure of change and/or the rate of change in thedynamic system, e.g., self-similarity. In some embodiments, similarityis determined between the selected image and all of the other images.

[0033] In some embodiments, the attribute is one or more of thefollowing: cell morphology, cell migration, cell motility, cell death(e.g., necrosis or apoptosis), cell division, binding to or interactingwith a second entity, organismal development, organismal motility,organismal morphological change, organismal reproduction, and themovement or morphological change of individual tissues or organs withinan organism.

[0034] In some embodiments, the method includes selecting a plurality ofimages and evaluating the similarity between pairs of images todetermine a pairwise similarity measurement, e.g., by computed pairwisecorrelations or by employing fourier optics; and generating aself-similarity matrix comprising the pairwise similarity measurements.

[0035] In some embodiments, the method includes selecting a plurality ofthe images and evaluating the similarity between all the images todetermine a pairwise similarity measurement, e.g., by computed pairwisecorrelations or by employing fourier optics, and generating aself-similarity matrix comprising the pairwise similarity measurements.

[0036] In another aspect, the invention provides methods for evaluatingan attribute of a dynamic biological system over time. The methodincludes some or all, typically all of the following (the steps need notbe but typically are performed in the order provided herein): providinga plurality of images representative of the dynamic biological system;generating a motion field from at least two images; and characterizingthe dynamic biological system as a statistical function of the motionfield, thereby evaluating the dynamic biological system. In someembodiments, the dynamic biological system is characterized using astatistical analysis of motion vectors, by evaluating direction and/orvelocity in the dynamic biological system, and/or by determining thedistribution of direction or velocity in the dynamic biological system.

[0037] In some embodiments, the method includes performing a statisticalanalysis of velocity as a function of direction and/or a statisticalanalysis of direction as a function of velocity.

[0038] In some embodiments, the method includes detecting one or moremoving objects, e.g., biological units, in the image, e.g., based onmotion vector colocomotion. In some embodiments, the method includesdetermining the direction or velocity of the moving object as a functionof colocomoting motion vectors. In some embodiments, the method includesperforming a statistical analysis of velocity as a function of directionand/or a statistical analysis of direction as a function of velocity.

[0039] In some embodiments, the method also includes determining thecenter of motion for a moving object. In some embodiments, the methodincludes determining the directional persistence of the moving object,determining the direction or velocity of the center of motion of themoving object, and determining the direction and velocity of the centerof motion of the moving object. The method can also include performing astatistical analysis of velocity as a function of direction, and/orstatistical analysis of direction as a function of velocity. In someembodiments, the method also includes determining the distribution ofdirection or velocity of a moving object.

[0040] In some embodiments, the method also includes establishing abounding box for a moving object, e.g., for each moving object. Thebounding box can correspond exactly to the maximum dimensions of theobject. The bounding box can correspond to the maximum dimensions of theobject plus a preselected factor. The size of the bounding box can varywith the self-similarity of the object. In some embodiments, the methodalso includes analyzing the area within the bounding box, e.g., byapplying image segmentation based on raw intensity, texture, and/orfrequency. In some embodiments, the method also includes evaluating anattribute of the object.

[0041] In some embodiments, the method also includes evaluating anattribute of the object, for example, by a method including some or all,typically all, of the following: providing a plurality of images of theobject; evaluating the similarity between a plurality of images of theobject; and characterizing the object as a function of the similaritybetween the images, e.g., by generating a self-similarity matrix. Insome embodiments, the images of the object are acquired by a methodcomprising acquiring images at a first acquisition parameterization;determining similarity between a selected image and at least one of theother images; characterizing the images as a statistical function ofself-similarity; and the acquisition parameterization is adjusted as afunction of the self-similarity of the images. In some embodiments, theplurality of images is a pair of images.

[0042] In some embodiments, the plurality of images of the objectcomprises images of the area within the bounding box. In someembodiments, the method also includes calculating the dimensions of theobject, e.g., a major axis and a minor axis. In some embodiments, themethod also includes characterizing the shape of the object as afunction of the major axis and the minor axis and/or generatingeigenvalues.

[0043] Any of the methods described herein can be applied to thecharacterization of a dynamic biological system. Accordingly, in anotheraspect, the invention provides methods for characterizing a dynamicbiological system comprising a biological unit, e.g., a plurality ofbiological units, e.g., independently selected from one or more ofcells, tissue, organs, and unicellular organisms, multicellularorganisms. The method includes some or all, typically all, of thefollowing (the steps need not be but typically are performed in theorder provided herein): providing the dynamic biological system;acquiring a plurality of images representative of the dynamic biologicalsystem in two dimensions; determining self-similarity between arepresentative set of the images; and characterizing the set of imagesas a statistical function of self-similarity, thereby characterizing thedynamic biological system. In some embodiments, the images are acquiredby a method comprising acquiring images at an initial acquisition and/orstorage parameterization; determining similarity between selectedimages; characterizing the selected images as a statistical function ofself-similarity; optionally comparing the characterization with areference value; and optionally adjusting the acquisition or storageparameterization as a function of the self-similarity of the images. Insome embodiments, the pre-selected reference value is a measure ofchange and/or the rate of change in the dynamic system, e.g.,self-similarity. In some embodiments, the plurality of images is a pairof images.

[0044] In some embodiments, the method provides information regardingone or more attributes of the biological unit. In some embodiments, thebiological unit is a cell, and in some embodiments, the one or moreattributes can be cell motility, cell morphology, cell division, celladherence. In some embodiments, the biological unit is an organism. Insome embodiments, the one or more attributes can be organismal motility,organismal morphological change, organismal reproduction, and themovement or morphological change of individual tissues or organs withinan organism.

[0045] In some embodiments, the dynamic biological system ismanipulated, e.g., by altering temperature, viscosity, shear stress,cell density, composition of media or surfaces contacted, electricalcharge, gene expression, protein expression, adding one or more otherbiological units of the same or different type, or by adding or removingor one or more treatments. In some embodiments, the manipulation isaddition or removal of a treatment, e.g., one or more test compounds,e.g., small molecules, nucleic acids, proteins, antibodies, sugars andlipids. In some embodiments, a plurality of dynamic biological system iseach exposed to a different manipulation. In some embodiments, aredundant set of dynamic biological systems is exposed to a redundantset of manipulations; for example, if a first set includes six dynamicbiological systems, and the six dynamic biological systems are eachexposed to a different manipulation, a redundant set would be a secondset of six dynamic biological systems exposed to the same sixmanipulations as the first set, resulting in the exposure of two dynamicbiological systems to each test compound.

[0046] In some embodiments, the method includes acquiring a plurality ofimages representative of the dynamic biological system at one or more ofthe following points: prior to, concurrently with, and subsequent to themanipulation. In some embodiments, the method includes evaluating theeffect of a manipulation, e.g., a treatment, on one or more attributesof the one or more biological units.

[0047] The methods of the invention can be combined with other methodsof evaluating a dynamic biological system, e.g., the effect of one ormore drug candidates on a dynamic biological system can be analyzed by amethod described herein in combination with a second method. The secondmethod can be a method of the invention or another method. The methodscan be applied in any order, e.g., a method of the invention can be usedto confirm a “hit” candidate compound identified in a prior screen whichdoes not use a method of the invention.

[0048] In some embodiments, the biological unit is a cell. In someembodiments, the attribute can be cell motility, cell morphologicalchange, cell adherence, and cell division.

[0049] In some embodiments, the biological unit is an organism. In someembodiments, the attribute can be consisting of organismal motility,organismal morphological change, organismal reproduction, and themovement or morphological change of individual tissues or organs withinan organism.

[0050] In some embodiments, the dynamic biological system includes aplurality of biological units that are all similar or include two ormore different biological units. The biological units can differgenetically e.g., as a result of gene deletion or duplication, targetedmutation, random mutation, introduction of additional genetic material,epigenetically, phenotypically or in developmental stage. The biologicalunits can also differ as a result of exposure to a manipulation, e.g., atreatment, e.g., a test compound.

[0051] In some embodiments, the method also includes evaluating theeffect of the manipulation on an attribute of a biological unit, andselecting the manipulation for further analysis. The further analysiscan be by a method described herein, or by a different method, e.g., amethod other than a method of evaluating a dynamic biological systemcomprising providing the biological unit; acquiring a plurality ofimages representative of the dynamic system in two dimensions;determining self-similarity between a representative set of images; andcharacterizing the images as a statistical function of self-similarity.

[0052] In some embodiments, wherein the manipulation is the addition orremoval of a treatment, the further analysis can be by a high throughputor parallel screen, e.g., a screen wherein a number of dynamicbiological systems, e.g., at least 10, 10², 10³, 10⁴, 10⁵, 10⁶, 10⁷,10⁸, 10⁹, 10¹⁰ or more are manipulated, e.g., exposed to a treatmentsuch as a test compound, e.g., a candidate drug, e.g., a candidate forinhibition or promotion of an attribute, e.g., at least 10, 10², 10³,10⁴, 10⁵, 10⁶, 10⁷, 10⁸, 10⁹, 10¹⁰ or more different manipulations,e.g., treatments, e.g., test compounds. Thus, in one example, each of aplurality, e.g., at least 10, 10², 10³, 10⁴, 10⁵, 10⁶, 10⁷, 10⁹, 10¹⁰similar dynamic biological systems, e.g., comprising cells, are exposedto a different test compound, e.g., a different chemical compound. Thetest compound can come from any source, including various types oflibraries, including random or nonrandom small molecule, peptide ornucleic acid libraries or libraries of other compounds, e.g.,combinatorially produced libraries. In many cases as discussed above aplurality of the same or similar dynamic biological systems and manydifferent drug candidates are tested, or alternatively different dynamicbiological systems, e.g., genetically different, e.g., mutants, are testwith a single drug. The screen can be for evaluating a test compound forits ability to interact with a biological unit, receptor or othertarget, e.g., a screen is selected based on combinatorial chemistry,computer-based structural modeling and rational drug design, determiningthe binding affinity of the test compound, phage display, and drugwestern. Such screens can comprise contacting a plurality of members ofa library, e.g., a library of compounds having variant chemicalstructures, with a plurality of dynamic biological systems and selectinga library member having a preselected property, e.g., the ability toaffect an attribute of a biological unit.

[0053] In some embodiments, the manipulation, e.g., a treatment, e.g., atest compound, was identified in prior screen, e.g., a screen performedprior to the method of the present invention. The prior screen can be bya method described herein or by a different method, e.g., a method otherthan a method of evaluating a dynamic biological system comprisingproviding the biological unit; acquiring a plurality of imagesrepresentative of the dynamic system in two dimensions; determiningself-similarity between a representative set of images; andcharacterizing the images as a statistical function of self-similarity.Examples of such screens include those which are based on binding of aligand to a target.

[0054] In another aspect, the invention provides methods for optimizingthe effect of a test compound on an attribute of a biological unit. Themethod includes some or all, typically all, of the following (the stepsneed not be but typically are performed in the order provided herein):selecting a first test compound; exposing a dynamic biological system tothe first test compound; acquiring a plurality of images representativeof the dynamic biological system in two dimensions; determiningself-similarity between a representative set of the images;characterizing the set of images as a statistical function ofself-similarity; providing a next generation test compound; exposing adynamic biological system to the next generation test compound;acquiring a plurality of images representative of the dynamic biologicalsystem in two dimensions; determining similarity between arepresentative set of the images; and characterizing the set of imagesas a statistical function of self-similarity. The activity of the firstand the next generation compound can be compared, e.g., with one anotherof with reference value to evaluate the compound. The steps of themethod can be repeated with successive next generation compounds, e.g.,to optimize the structure of a test compound, e.g., to maximize theeffect of the test compound on an attribute.

[0055] In some embodiments, the first test compound and the nextgeneration compound are selected from a database of compounds of knownchemical structure. In some embodiments, the next generation compound isa variant, e.g., a structural variant, of the first test compound. Forexample, a particular moiety or functional group can be altered once orserially to identify optimized structures. In some embodiments, morethan one moiety or functional groups can be varies, simultaneously orserially.

[0056] In some embodiments, the method also includes selecting a firsttreatment; providing a next generation treatment; exposing a dynamicbiological system to the next generation treatment; acquiring aplurality of images representative of the dynamic biological system intwo dimensions; determining self-similarity between a representative setof the images; and characterizing the plurality of images as astatistical function of self-similarity.

[0057] In some embodiments, the method also includes acquiring aplurality of images representative of the dynamic biological system atone or more of the following points: prior to, concurrently with, andsubsequent to the exposure to the next generation treatment.

[0058] In another aspect, the invention also provides a method, e.g., amachine-based method, for determining the relationship between aproperty of a test compound, or a series of test compounds, and theability to modulate an attribute of a biological unit. The methodincludes some or all, typically all, of the following (the steps neednot be but typically are performed in the order provided herein):providing a first test compound having a first property, e.g., a firstchemical structure or property, e.g., a first moiety or structural groupat a selected position; exposing a dynamic biological system comprisinga biological unit to the first test compound; analyzing the dynamicbiological system by a method described herein, e.g., by acquiring aplurality of images representative of the dynamic biological system intwo dimensions; determining self-similarity between a representative setof the images; characterizing the set of images as a statisticalfunction of self-similarity; providing a second test compound having atleast one property similar to a property of the first treatment and atleast one property that differs, e.g., a moiety or functional group,e.g., an R group is varied between the first and second compound;exposing a dynamic biological system comprising a biological unit to thesecond test compound; analyzing the dynamic biological system by amethod described herein, e.g., by acquiring a plurality of imagesrepresentative of the dynamic biological system in two dimensions;determining self-similarity between a representative set of the images;characterizing the set of images as a statistical function ofself-similarity; and correlating the similar property of the first andsecond test compounds with an effect on one or more attribute.

[0059] In some embodiments, the property of the test compound isselected from the group consisting of chemical structure, nucleic acidsequence, amino acid sequence, phosphorylation, methylation, sulfation,nitrosylation, oxidation, reduction, affinity, carbohydrate structure,lipid structure, charge, size, bulk, isomerization; enantiomerization;and rotational property of a selected moiety, or any physical orchemical property of the structure. For example, a moiety is present ona scaffold and the moiety is varied allowing analysis of the ability ofthe moiety, or other moiety at the same position, to affect anattribute.

[0060] In another aspect, the present invention provides a method, e.g.,a machine-based method, for evaluating or selecting a target, e.g., tomediate a selected attribute of a biological unit. The method includessome or all, typically all of the following (the steps need not be buttypically are performed in the order provided herein): providing a firsttest compound, e.g., a ligand, for a first target, e.g., a receptor;contacting a dynamic biological system comprising a biological unit withthe first test compound; and performing a method described herein, e.g.,a method including: (1) acquiring a plurality of images representativeof the dynamic biological system in two dimensions; (2) determiningself-similarity between a representative set of the images; and (3)characterizing the set of images as a statistical function ofself-similarity; thereby providing a value for a parameter related tothe effect of the first test compound on the selected attribute;providing a second test compound, e.g., a ligand, for a second target,e.g., a different receptor; contacting one or more biological units withthe second test compound; and performing a method a method describedherein, e.g., a method including: (1) acquiring a plurality of imagesrepresentative of the dynamic biological system in two dimensions; (2)determining self-similarity between a representative set of the images;and (3) characterizing the set of images as a statistical function ofself-similarity, thereby providing a value for a parameter related tothe effect of the second test compound on the selected attribute; andcomparing the parameters and selecting the test compound having thedesired effect on the attribute, thereby selecting a target.

[0061] In one aspect, the invention provides a method, e.g., amachine-based method, for evaluating the activity of a gene. The methodincludes some or all, typically all, of the following (the steps neednot be but typically are performed in the order provided herein): themethod comprising: providing a first reference biological unit orplurality thereof; providing a second biological unit or pluralitythereof wherein the activity of the gene is modulated as compared to thefirst biological unit, and performing a method described herein, e.g., amethod comprising: (1) acquiring a plurality of images representative ofthe dynamic biological system in two dimensions; (2) determiningself-similarity between a representative set of the images; and (3)characterizing the set of images as a statistical function ofself-similarity, thereby evaluating the activity of the gene.

[0062] In some embodiments, the gene is modulated by directed or randommutagenesis. In some embodiments, a plurality of genes are modulated,e.g., by random mutagenesis. In some embodiment, the plurality of genesare selected from the results of an expression profile experiment, e.g.,a gene chip experiment or are expressed in or known to be associatedwith a disease state.

[0063] In some embodiments, the plurality of genes are modulated in aplurality of biological units and/or dynamic systems. In someembodiments, a unique gene is modulated in each of a plurality ofbiological units and/or dynamic systems.

[0064] In some embodiments, the method includes manipulating the dynamicsystem and evaluating the effect of the manipulation on the activity ofthe gene.

[0065] In another aspect, the invention provides a method, e.g., amachine-based method, of evaluating the interaction of a biological unitwith a surface. The method includes some or all, typically all, of thefollowing (the steps need not be but typically are performed in theorder provided herein): providing a dynamic biological system comprisinga biological unit; contacting the dynamic biological system with asurface; and performing a method described herein, e.g., a methodcomprising: (1) acquiring a plurality of images representative of thedynamic biological system in two dimensions; (2) determiningself-similarity between a representative set of the images; and (3)characterizing the set of images as a statistical function ofself-similarity, thereby evaluating the interaction of the biologicalunit with the surface.

[0066] In some embodiments, the surface is uniform. In some embodiments,the surface is variable, e.g., comprises pores, openings, concavities,convexities, smooth areas, and rough areas, changes in composition,changes in charge, and/or the presence or absence of a test compound,e.g., the test compound is present in a gradient.

[0067] In some embodiments, the interaction can be adherence to thesurface, movement across the surface, release from the surface; depositor removal of a material on the surface, and infiltration of pores oropenings.

[0068] In another aspect, the invention provides a method, e.g., amachine-based method, for evaluating the propensity of one or morebiological units to interact with, e.g., infiltrate a structure, e.g.,the surface of a prosthetic device, e.g., stainless steel, titanium,ceramic, and synthetic polymer. The method includes some or all,typically all, of the following (the steps need not be but typically areperformed in the order provided herein): providing one or morebiological units; providing a structure, e.g., a piece of a prostheticdevice; performing a method comprising: (1) acquiring a plurality ofimages representative of the dynamic biological system in twodimensions; (2) determining self-similarity between a representative setof the images; and (3) characterizing the set of images as a statisticalfunction of self-similarity; thereby evaluating the propensity of thebiological units to infiltrate a structure. In some embodiments, themethod also includes exposing the biological units to a test compoundand evaluating the effect of the test compound on the propensity of thebiological units to infiltrate the structure.

[0069] In another aspect, the invention provides a method, e.g., amachine-based method of evaluating the interaction between a biologicalunit and a second entity, e.g., bone cells, tissues, e.g., transplanttissue, e.g., allogeneic, autologous, or xenogeneic tissue. In someembodiments, the method includes providing one or more biological units;providing a second entity; performing a method described herein, e.g., amethod including: comprising: (1) acquiring a plurality of imagesrepresentative of the dynamic biological system in two dimensions; (2)determining self-similarity between a representative set of the images;and (3) characterizing the set of images as a statistical function ofself-similarity, thereby evaluating the interaction of the biologicalunits and the second entity.

[0070] In another aspect, the invention provides a method, e.g., amachine-based method, of evaluating a test compound. The method includessome or all, typically all of the following: providing a firstbiological unit; providing a second biological unit that is the same asthe first biological unit or plurality thereof wherein the first andsecond biological units are preferably the same; contacting the secondbiological agent with the test compound; performing a method describedherein, e.g., a method including: acquiring a plurality of imagesrepresentative of the dynamic biological system in two dimensions;determining self-similarity between a representative set of the images;and characterizing the set of images as a statistical function ofself-similarity; and comparing the attributes of the biological unit inthe presence and absence of the test compound, thereby evaluating thetest compound. In some embodiments, the method also includes: providinga second test compound; contacting the first biological unit with thesecond test compound; performing a method described herein, e.g., amethod including: (1) acquiring a plurality of images representative ofthe dynamic biological system in two dimensions; (2) determiningself-similarity between a representative set of the images; and (3)characterizing the set of images as a statistical function ofself-similarity; and comparing the attributes of the biological unit inthe presence and absence of the test compound.

[0071] The invention includes the systems and apparatus describedherein. Accordingly, in one aspect, the invention provides an apparatusthat includes some or all, typically all, of the following: anacquisition system, e.g., a sensor, configured to acquire images, e.g.,spatiotemporal or other signals, representative of a dynamic system atan adjustable parameterization; a storage device configured to store theimages at an adjustable parameterization; and a computing deviceconfigured to analyze similarities between the images (e.g., imagesacquired by the acquisition system). The apparatus can also include adisplay device. In some embodiments, the apparatus also includesbuffering means for potential processing of one or more images.

[0072] In some embodiments, the computing device is further configuredto adjust the acquisition parameterization of the acquisition deviceand/or the storage parameterization of the storage device as astatistical function of the similarity between images, e.g., includessetting the acquisition parameterization to drive the statisticalfunction to a predetermined level, e.g., setting the acquisition and/orstorage parameterization so that at least one or more most recentlyacquired images reflects a predetermined rate of change, e.g., settingthe acquisition parameterization so that at least one or more mostrecently acquired images reflects a predetermined rate of motion, shapechange, focal change, temperature change, or intensity change. Theacquisition parameterization can be, but is not limited to, acquisitionrate, exposure, aperture, focus, binning, or other parameter. Thestorage parameterization can be, but is not limited to, image labeling.

[0073] In another aspect, the invention features a database. Thedatabase includes a plurality of records wherein each record includes atleast one of the following:

[0074] a. data on the identity of a biological unit;

[0075] b. data on an attribute of the biological unit; and

[0076] c. data on a the effect of one or more manipulation, e.g., atreatment, e.g., the administration of a test compound, on theattribute.

[0077] In some embodiments, the data on the identity of the biologicalunit includes genotypic and phenotypic information, e.g, informationregarding the presence, absence, spatial location, or temporalexpression of a gene, and/or information regarding the presence orabsence of one or more mutations.

[0078] In some embodiments, the phenotypic data includes one or more ofcell type, organism type, cell status, and age.

[0079] In some embodiments, the database includes at least two records,and the manipulation in each of the records differs from the otherrecord. In some embodiments, the manipulation is administration of atest compound and in one record the preselected factor includesadministration of the test compound and in the other record the testcompound is not administered or is administered at a different dose. Insome embodiments, the database includes at least two records, and atleast one manipulation in each of the records differs from the otherrecord. In some embodiments, at least one manipulation in the recordsdiffers and at least one of the other manipulations is the same.

[0080] In another aspect, the invention provides a method foridentifying an unknown target, e.g., a gene, protein or other cellularor extracellular target. The method includes some or all, typically all,of the following: providing a database described herein, including atleast a first record having data about the effect of a firstmanipulation on a attribute, where the target of the first test compoundis known; and at least a second record having data about the effect of asecond manipulation on an attribute, where the target of the secondmanipulation is unknown; and comparing the data in the first record tothe data of the second record.

[0081] In some embodiments, the database is in computer readable form.

[0082] Methods and apparatus are described herein to assessself-similarity over the entire length of a spatiotemporal signal aswell as on a moving temporal window. In one aspect, a real time signalacquisition system is provided in which, self-similarity in a movingtemporal window enables adaptive control of acquisition, processing,indexing, and storage of the said spatiotemporal signal. In anotheraspect, such system as provided in which self-similarity in a movingtemporal window provides means for detecting unexpected. In yet anotheraspect such system as provided in which, self-similarity in over theentire length of a spatiotemporal signal or a moving or stationarywindow, provides means to characterize, classify, and compare dynamicprocesses viewed.

[0083] A method for measuring self-similarity of a spatiotemporal signalin systems according to some aspects of the invention includes steps ofassuring and maintaining of acquisition at or near the rate of dominantmotion in the visual scene as to assure as near linear relationshipbetween any two successive frames or times of acquisition. Furtherprocessing includes comparison of near and long range, distance in time,frames. Said comparisons for a temporal window, length greater than onecan be arranged in a matrix arrangement. In accordance to furtheraspects of this invention the said matrix is used to computeself-similarity for the respective temporal window.

[0084] Further aspects of the invention provide such methods andapparatus that use an unsupervised learning algorithm to classifystatistical dependence of one or more sections of the acquiredspatiotemporal signal on any other section of said signal uncoveringperiodic, regularities, or irregularities in the scene. Said algorithmcan be unsupervised insofar as it requires no tuned or specific templateor noise model to measure self-similarity and thus describe the visualdynamics.

[0085] Further aspects of the invention provide for efficient, costeffective and salient computation of cross matches between framesseparated by long range of temporal distance by utilizing the persistedoperated model of linearity or near linearity of successive frames andgeometric mean of cross-matches in the frequency domain.

[0086] Further aspects of the invention provide such methods andapparatus that prescribe an efficient, cost effective, and salientmeasurement of visual self-similarity across indefinitely longacquisition duration. Visual self-similarity measured, according torelated aspects of the invention, can be used to characterize, quantify,and compare the underlying dynamic system to the best representation ofthe its visual projection.

[0087] Further aspects of the invention provide automatic methods forrecording of exemplary templates of a acquisition session. A frame islabelled an exemplary template when it is kernel of a sequence of framesacquired consecutively whose incremental temporal integration forms alinear set with any and every frame in the sequence. A related aspect ofthis invention is its providing means to recognize novel andunpredictable frame or sequence of frames by their nonlinearrelationship with the rest of the acquired frames.

[0088] Related aspects of the invention provide such methods andapparatus that provide predictive feedback to the acquisition sub-systemas to appropriateness of the parameters) controlling temporal sampling,e.g., in the case of video acquisition, typically, frame-rate andexposure.

[0089] An information theoretic mechanism can be used, according tostill further aspects of the invention, to compute whole orself-symmetry measurement for a group of frames in the buffer. The wholecharacterization, according to related aspects of the invention, can betracked and matched to characterizations generating a predictive signalfor adjustment of acquisition parameters for frame-rate and exposure aswell as identifying frame sequences of interest.

[0090] Further aspects of this invention provide such methods andapparatus that prescribe Fourier optics system for computation of crossmatches between successive frames.

[0091] Further aspects of the invention provide methods and apparatus asdescribed above that utilize conventional or other acquisition devicesto measure motion signatures indicating speed and type of dominantcharacterizing motion in view.

[0092] Still further aspects of the invention provide such methods andas are operationally customized via a script that encapsulates usersstorage and indexing preferences.

[0093] Unless otherwise defined, all technical and scientific terms usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which this invention belongs. Although methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of the present invention, suitable methods andmaterials are described below. All publications, patent applications,patents, and other references mentioned herein are incorporated byreference in their entirety. In case of conflict, the presentspecification, including definitions, will control. In addition, thematerials, methods, and examples are illustrative only and not intendedto be limiting.

[0094] Other features and advantages of the. invention will be apparentfrom the following detailed description, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0095]FIG. 1 is a block diagram of an embodiment of the apparatus.

[0096]FIG. 2A is a high-level block diagram of an embodiment of themethod.

[0097]FIG. 2B is a detailed block diagram of an embodiment of themethod.

[0098]FIG. 3 is detailed block diagram of an embodiment of the analysismodule.

[0099]FIG. 4A is a block diagram of attentive capture initialization.

[0100]FIG. 4B is a block diagram of method of the self-similaritycalculation.

[0101]FIG. 5 is a diagram of a self-similarity matrix and entropicindices.

[0102]FIG. 6 is a diagram of the use of overlapping tiles for motionestimation.

[0103]FIG. 7 is a block diagram of the method of global motionestimation.

[0104]FIG. 8 is a block diagram of the method of estimatingself-similarity.

[0105]FIG. 9 is a schematic diagram of the method of estimatingself-similarity.

[0106]FIG. 10 is diagram of the overlap between cellular attributes andtherapeutic areas.

DETAILED DESCRIPTION

[0107] The present invention provides methods and apparatus forcharacterizing dynamic systems. Embodiments of the invention are furtherdescribed in the following description and examples, which do not limitthe scope of the invention described in the claims.

[0108] A block diagram of an embodiment of the apparatus foracquisition, compression and characterization of spatiotemporal signalsincludes a sensor(s) (102), data processing device(s) (also known ascomputing device(s)) (103), storage device(s) (105) and display (104)devices as shown in FIG. 1.

[0109] Data processing device(s) (103) includes one or more modules(fabricated in software, hardware or a combination thereof) executing onone or more general or special purpose digital data processing or signalprocessing device(s) in accordance with the teachings below.

[0110] The sensor (102) can be one or more video cameras (of theconventional variety or otherwise) or other devices suitable foracquiring spatiotemporal, thermal or other signals (regardless ofwhether those signals represent the visible light spectrum)representative of a system to be subjected to characterization, indexingor other processing in accordance with the teachings hereof. In oneembodiment, the sensor can be monitoring a dynamic system as definedbelow. However, the teachings herein may also be applied to themonitoring of a non-dynamic system, such as in cases where a system isthought to have the potential to be dynamic, or when a comparison is tobe made between systems where at least one system is thought to have thepotential to be dynamic.

[0111] The sensor can be parameterized or tuned to receive a particularband or bands of frequency, such as might be required, by way ofexample, for fluorescent imaging techniques. Suitable devices (109) canbe inserted between the scene (101) and the sensor to amplify, magnify,or filter or otherwise manipulate the information in the scene prior toits acquisition by the sensor. The output of the sensor (107) isreferred to hereafter as an “image” or a “frame,” regardless of the typeof sensor and whether or not the image is a direct representation ofsensory data, reconstructed sensory data or synthetic data. Element 102can alternatively be a source of previously acquired video,spatiotemporal or other signals representative of a dynamic system. Forthe sake of convenience and without loss of generality, element 102 ishereafter referred to as “sensor.”

[0112] The sensor (102) can also be, by way of non-limiting example, asource for a multitude of stored frames in two or more dimensions, suchas a collection of photographic images, and an embodiment of the presentinvention can be used to cluster said frames into classes correspondingto measurements of self-similarity, regardless of whether any or all ofthe frames were acquired from the same system or scene.

[0113] Element 102 can also be, by way of further non-limiting examples,two or more cameras or other sensors in a stereo or other multi-sourceimage acquisition system; one or more sensors that include one or morefiltering devices between the scene and the signal acquisition device;or an ensemble of sensory modalities, each represented by one or moresensors.

[0114] A dynamic system is defined as a system in which values output bya sensor monitoring the system vary across time. A dynamic system can bea system that is “naturally” dynamic, i.e., a system that changeswithout external perturbation, and would most commonly be viewed by astationary sensor. A dividing cell, for example, would be a dynamicsystem. However, a system can be induced to produce varying output fromthe sensor through a variety of means, including: perturbing ormanipulating an otherwise non-changing system being monitored by astationary sensor, such as would happen when positioning and orienting asemiconductor wafer for photolithography or placing a chemoattractantnear a stationary cell; perturbing a sensor that is monitoring anon-changing system, such as would happen when panning a video cameraover a document or large photograph; perturbing the signal prior to itsoutput by the sensor through electronic, programmatic or other means; orany combination of perturbations and natural dynamism that would lead tovariance in output from the sensor. For the sake of convenience, imagesare said to be representative of a dynamic system, or particularly adynamic system over time, regardless of whether the system is inherentlydynamic or made to appear dynamic by virtue of imaging modality or anyinduced perturbation.

[0115] Images can be processed before analysis. Processing can includefiltering, windowing, sub-sampling, integration, integration of thesquares and gradient detection. Images, processed or unprocessed, willbe referred to hereafter simply as “images” or “frames”. Images andframes are represented by an array of values representing intensity. Aframe or image sequence (106) is a set of arrays of values representingsensory information, where each frame is or could be related to everyother frame in some way. In some embodiments, this relationship may beby virtue of the fact that the frames were acquired sequentially by asingle sensor, though in other modes this relationship may be throughthe sharing of similarities in shape, color, frequency or any of anumber of other attributes. The sequence may also be defined throughordered or random selection of a subset of frames from a larger set.Frame rate defines the number of frames captured in a unit of time.Exposure time is the length of time a sensor is exposed to the scene(101) while acquiring the data that produces a single frame. Frame rateand exposure time have their usual definitions in the field of visualsignal processing. Other sensory modalities have analogous variables.

[0116] The illustrated reporting module (203) is comprised of storagemedia (dynamic, static or otherwise) with suitable capacity for at leasttemporary storage of video or other spatiotemporal sequences that may beacquired, compressed, characterized and/or indexed by the illustratedembodiment. In FIG. 1, by way of non-limiting example, the storagedevice (105) is depicted as a disk drive.

[0117] The acquisition process starts with the establishment of aninitial or first acquisition rate and an attention window (108) size.These parameters can be specified manually or programmatically, based onsystem capabilities, empirical knowledge about the sensor or the scene,or through other means. The “attention window” is a frame sequence whoselength is specified in units of time or some other relevant metric, suchas number of frames or interval between peaks in a particular function.One use of the attention window in the present invention is forcomputing relationships between “short-term” frames, e.g., frames thatare close to each other based on measurements of acquisition time,similarity or other metrics. In some embodiments, a maximum frame rateand the corresponding frame size in memory are also derived from thesystem information. By way of non-limiting example, the attention windowsize for processing video images representative of cell spreading canrange from ½ to many seconds, though other sizes may be used forcapturing this and other processes. When an acquisition subsystem isreplaced with a signal source, maximum frame rate is preferably theframe rate at which the data was acquired.

[0118] In some embodiments, the analysis module contains afirst-in-first-out (FIFO) frame sequence buffer, though other bufferingdesigns are possible. Preferably, this buffer is maintained in ahigh-speed storage area on the data processing device, such as a desktopcomputer's random access memory, though storage on a disk drive or otherdigital medium is also possible. In a preferred mode, the frame sequencebuffer is sized according to the mathematical relation buffersize=(attention window size in seconds*initial frame rate inseconds*memory space needed for each frames)+an overhead factor. Theoverhead factor is selected empirically and, for example, can be in therange 1 to 5 percent, depending on memory management design. By way ofnon-limiting example, a frame sequence buffer for processing videoimages representative of a biological process may range from 30 to 120MBytes, though other sizes may be used for these and other processes.Frames in the FIFO may also represent a fixed or variable or adaptivelyvariable sampling of the incoming acquired frames. Incoming framesoriginate at the sensor (102). Frames exiting the data processing device(103) for storage or display (105 and 104) have associated labels andcharacterization data attached to them.

[0119] In some embodiments, frames are also optionally prefiltered tosuppress or promote application-specific spatiotemporal characteristics.Incoming frames, by way of example, could be processed by standardmethods in the art to extract gradient direction estimation at aparticular spatial scale to amplify signals representative of changes indirection of an object or organism moving in the scene.

[0120] In some embodiments, certain analyses are performed on theluminance channel of each frame. In other embodiments, multiple colorchannels within each frame can be matched separately to correspondingcolor channels in other frames, with the resulting values combined intoa single set of measurements or presented as distinct, one set per colorchannel. Still other embodiments incorporating visual sensors may useother channels in addition or instead of these, and embodimentsincorporating non-visual sensors would use channels appropriate to theinformation produced by the sensor.

[0121] In some embodiments, certain analyses are performed on thedominant frequency band in each frame. This is a preferred mode when theassumption can hold that frequency content changes minimally betweensuccessive frames. The choice of frequency band(s) analyzed in otherembodiments may be influenced by other factors.

[0122] In some embodiments, certain analyses are performed viacorrelations between sets of individual frames. In other embodiments, aframe might be correlated with a temporal accumulation or per-pixel rankfunction of some number of related frames. Many variations on thischoice for the present embodiment and others noted above, includingchoices regarding how to process chromatic channels, regions of framesused, and potential preprocessing steps can be implemented to producesimilar results.

[0123] Frames are transferred into the frame sequence buffer from thesensor (102) in a conventional manner. As widely known in the art,references to said frames can be used to remove the need to utilizesystem resources for an image copy.

[0124] Next, spatiotemporal signals in the acquired frames are analyzed.It is well-known in the art, by way of Parseval's Theorem that theintegral of a spatiotemporal signal over time is proportional to theintegral of its spatiotemporal frequencies.

[0125] ∫_(x)∫_(y)∫_(t) I(x,y,t)≈∫_(wx)∫_(wy)∫_(wt) F(w _(x) ,w _(y) ,w_(t))

[0126] Where I is intensity, F is frequency, x and y are spatialcoordinates, t is time, w_(x) and w_(y) are the frequency components inthe spatial dimensions and w_(t) is the frequency component in thetemporal dimension.

[0127] Put another way, the integral of the spatiotemporal signalbetween time t (0→t) and t+n (0→(t+n) is an estimate of the change inspatiotemporal frequencies from time t to (t+n). When frames areacquired at a frame rate above the rate of change of the fastestchanging element in the scene, the acquired frames are nearly identical,the integral of the underlying signal approaches a constant value fromframe to frame, the difference in information between frames becomesnegligible, yet spatial definition within the frame remains high andinformation content is high. In contrast, when elements change fasterthan the frame rate of the sensor, the frames are blurred: the integralof the underlying signal also approaches a constant value from frame toframe, but frames lose their spatial definition and consequentlyinformation content is reduced. Thus, an estimate of information rate isdirectly proportional to the rate of change in the temporalautocorrelation function, and consequently in the integral of thespatiotemporal frequencies.

[0128] Methods known in the art can be used to estimate changes in therate of information content, though such estimates have limitations.Art-known compression standards such as MPEG are largely based on anassumption of fixed capture rate and output rate. MPEG encoders useblock-based motion calculations to discover temporally redundant databetween successive frames. This leads to the implementation of threeclasses of frames: spatially encoded frames (I), predicted frames (P)and bidirectional frames (B). Encoding frames in this way with ablock-based technique, and relying especially on predicted frames toenable efficient compression, leads to data loss that couldsignificantly impair the information content of frames that are foundsubsequently to be of particular interest. Furthermore, the MPEG methodestimates the rate of change in information content using coarse andnon-overlapping spatial blocks across a very narrow window (2-3 frames).This leads to further information loss. The net result is that MPEGcompression enables temporal integrity in compression and playback, butat the loss of spatial integrity. The present invention enables thepreservation of temporal integrity in frame sequences of interest, whilealso preserving spatial integrity.

[0129] Another compression standard, Motion JPEG, does not enabletemporal compression and instead applies a variant of the standardsingle-image JPEG compression to every frame. In Motion JPEGcompression, the rate of change in information content is estimated onlyspatially, and results in chromatic loss. Another approach, employingsimple motion detectors, uses the average intensity difference betweensubsequent frames as an estimate of the rate of change in informationcontent. This approach is limited in a number of ways, including that achange in lighting on a static scene would be perceived as “motion” eventhough nothing in the scene actually moved.

[0130] A human observer can easily and without any previous trainingmeasure attributes of a visual dynamic scene. This implies, for exampleand in a non-limiting way, that there may exist a significant amount ofmutual information that is implied and reinforced by each and everyframe in an attention window into a dynamic visual scene. In a dynamicsystem, events captured in closely-spaced frames and those captured indistant frames all impact the rate of change in information content. Byperforming the present methods in a preferred mode at or near the rateat which frame-to-frame information change is minimized, acharacteristic function of the dynamic system can be estimated in smalldiscrete steps where the values produced for a given frame depend onnearby frames as well as distant frames. By way of non-limiting example,such a system could be used to monitor a biological assay in which anevent of interest is a particular motion pattern of a nematode worm. Thepattern may last only fractions of a second, and may occur infrequentlyand unpredictably during the 18-day life of a wild-type worm.Nevertheless, moments in which this pattern was sensed would producereinforcing information over the worm's life, and periods of absence ofthis pattern would produce reinforcing information of its absence.Therefore, the difference between the two, as represented in aself-similarity function such as those in the present embodiment, wouldenable the automated detection of each instance of the event ofinterest. An early realization of importance of both short-term andlong-term correlations, as well as self-similarity as a model, was madeby Mandelbrot during his work on 800 years of flood data on the NileRiver during the construction of the Aswan Dam. Nevertheless, thoseskilled in the art have not yet found efficient methods to takeadvantage of long-term correlations in self-similarity analysis. Thepresent invention provides such methods.

[0131] Some embodiments of the invention use a Self-similarity matrixfor modeling and analyzing spatiotemporal signals, as shown in FIG. 2.In the illustrated embodiment, the self-similarity matrix is a squarematrix of normalized positive probabilities having a diagonal of ones,though in other embodiments the self-similarity matrix may have othercharacteristics instead or in addition. A self-similarity matrix has theform of a symmetric matrix, e.g., a real value Hermitian Matrix. In someembodiments, the invention employs a self-similarity matrix, frames andframe sequences to approximate a temporal autocorrelation of theacquired signal.

[0132] In some embodiments the invention exploits the near similarity offrames when sampled temporally at a rate near the dominant motion in thevisual scene to approximate an autocorrelation. The nearly similarframes in aggregate approximate a correlation of a given frame withslightly shifted versions of itself. In other embodiments, correlationscan be performed at a multitude of frequencies, or an autocorrelationfunction can be computed using methods well known in the arts.

[0133] Other embodiments of the invention might use other learning orapproximation algorithms. Popular methods for analyzing spatiotemporalsignal include PCA or ICA (Principal Component Analysis or IndependentComponent Analysis). In particular, PCA and ICA methods both employ acorrelation matrix and are widely used in lossy compression methods.

[0134] In the illustrated embodiment, the self-similarity matrix ispopulated with all pairwise similarity measurements. Other embodimentsmight measure pairwise dissimilarity. Such measurement isstraightforward to achieve within the present invention due to the factthat the sum of a similarity measurement and its correspondingdissimilarity measurement is always 1.0. Thus, (1—similaritymeasurement) yields the dissimilarity measurement. Known in the art isthat Fourier optics can also be used to produce pairwise correlationsbetween sequential frames as they are captured by a sensor. Framesgenerated in this way may be used for further analysis in accordancewith the teachings herein.

[0135] In some embodiments, the pairwise similarity metric chosen is anormalized correlation (multiplicative) applied to the entire frame. Theresult of this kind of cross-match is a scalar value from—1.0 (perfectmismatch) to 1.0 (perfect match). In the illustrated embodiment, forreasons described below, we use the square of the cross match. In anycase, the similarity metric is associative (Similarity (a,b)=Similarity(b,a)), Reflective (Similarity (a,a)=1.0), and Positive (Similarity(a)>0).

[0136] A well-known method for establishing image similarities is the“sum of absolute differences”. This method has both advantages anddisadvantages when compared to normalized correlation. Advantages tousing the sum of absolute differences include:

[0137] (a) It is often faster on many computer platforms, and

[0138] (b) It is well-defined on flat intensity patches.

[0139] Disadvantages Include:

[0140] (c) Cross-match result is not normalized,

[0141] (d) Cross-match result is not invariant to linear changes inintensity, and

[0142] (e) It is not equivalent to linear filtering.

[0143] In some embodiments, the present implementation of normalizedcorrelation takes advantage of modem computing architectures to achievenear-parity in computational performance with a “sum of absolutedifferences” approach, and also detects when the input images have zerovariance, thus enabling good definition on flat intensity patches.

[0144] In other embodiments, the cross-match operation can beaccomplished by other multiplicative, subtractive, feature-based orstatistical operations. In the illustrated embodiment, the similaritymeasurements have the additional property of behaving as spatiotemporalmatched filters. Yet another embodiment might use othercorrelation-based motion detectors.

[0145] The self-similarity estimator module (302) estimates short termtemporal similarity and approximates long term temporal similarity. Inbinocular applications, self-similarity is measured between each framefrom each camera and every frame acquired by the other camera. In yetother applications, integration of spatiotemporal signals or the squareof such signal may be used.

[0146] Short-term frames refer to frames in the above-mentioned buffer.Long-term frames refers to frame no longer in the buffer. The role ofself-similarity is twofold: first, to boost nearby frames that aresimilar, and second, to reduce the contribution of dissimilar frameselsewhere. Those skilled in the art may recognize the usage ofself-similarity in representing a nonlinear dynamic system or a dynamicsystem of unknown linearity. Self-similarity is estimated from the:

SS ₆₆=Self-Similarity Matrix (X, Δ)  (1)

[0147] where X is the time series, and Δ is the time duration over whichself-similarity is measured. In some embodiments, the self-similaritymatrix is a square matrix.

[0148] To estimate short-term self-similarity, similarity of all framesin the buffer can be measured.

SS _(short-term, Δ)=Self-Similarity Matrix (X, Δ)  (2)

[0149] where X is time sequence of frames, and Δ is the length of thebuffer, and

SM(i,j)=correlation(min(i,j), max(i,j)),  (3)

[0150] for all frames i, and j!=i (associativity)

SM(i,i)=1.0  (4)

[0151] (reflectivity)

[0152] In some embodiments, as frames are acquired and placed in theimage buffer (301), similarity matching is performed on at least themost recent frame and the frame immediately prior to it. Long-termpairwise matching between any two frames is approximated by treating thestring of pairwise correlations separating the frames as transitionalprobabilities. Similarity metrics other than those described hereincould be used, with an impact on the accuracy of this approximation.Correlation in the spatial domain is equivalent to a conjugatemultiplication in the frequency domain. In some embodiments,

SS _(long-term, Δ)=Self-SimilarityMatrix (X, Δ)  (5)

[0153] where X is a sequence of frames and Δ is the length of the FIFO,and

SM(i,j)=correlation(i,j),

[0154] for all i,j and distance (ij)=1(associativity)

SM(i,i)=1.0  (7)

[0155] (reflectivity)

SM(i,j)=(Π_(i->j) SM(i,i+1))^(1/(j−i)),  (8)

where j>(i+1)  (8A)

[0156] Equation (8) calculates the geometric mean of the pairwisecorrelation values separating i and j. Note that the approximations areassociative, degrade with distance between i, and j, and produce 0 whenany pairwise correlation along the way is 0. Further note thatapproximations are symmetric, SM(i,j)=SM(j,i).

[0157] Long-term and short-term similarities are combined to establish aself-similarity matrix for the entire duration of analysis. In someembodiments, lengthy durations may have windows of time where bothshort-term estimations and long-term approximations are used forsimilarity measurements. In some embodiments, shorter durations useshort-term estimations entirely. Typically, this choice would largely bebased on computational resources.

[0158] Further processing of the self-similarity matrix is independentof how the similarity measurements were produced; that is, themeasurements can be produced via short-term estimation, long termapproximation, or any weighted, normalized, or raw combination of thetwo.

[0159] In some embodiments, the self-similarity matrix (505) is thenused to estimate a measure of self-similarity for every frame with aframe sequence, as shown in FIG. 5. In some embodiments, an estimationof entropic indices for each frame is computed from the self-similaritymatrix (506). In some embodiments, by way of non-limiting example,Shannon's Entropy Calculation is used. Shannon's entropy calculates theaverage uncertainty removed by a random variable attaining a set ofmeasurements.

P _(j) =SM _(j)/Σ_(j) SM _(i,j)  (9)

[0160] normalization

H _(j) =−ΣP _(j) log ₂(P _(j))/log2(n)  (10)

[0161] where n is number of frames

[0162] For a given sequence, a random variable represents a set ofentropic indices for each frame in the sequence. If all the frames inthe sequence are exact copies, uncertainty is completely removed andShannon's entropy is nearly zero. On the other hand, if every frame iscompletely dissimilar to all other frames, no uncertainty is removed andShannon's entropy is nearly one.

[0163] The self-similarity matrix can be evaluated over the length of aframe sequence where said sequence can be fixed or can be a slidingwindow across a larger sequence. The calculation of self-similarity forsaid sliding window uses in preferred mode standard and well-knownoptimizations to reduce the computational cost to a linear function ofthe number of frames in the frame sequence.

[0164] Existing methods can meaningfully quantifying events of interestin images and image sequences, but only after a spatial or temporalsegmentation step. In most cases, these steps are costly in terms ofcomputational time and human intervention, and are impaired by thenatural occurrence of noise in the signal. In some embodiments of theinvention, dynamic systems presenting events of interest withcharacteristic visual signatures can be quantified without temporal orspatial segmentation. An example is a spatiotemporal signal representinga visual focusing process. Frames from said signal, as an example, mayrepresent temporally out-of-focus frames, increasingly sharper frames,and in-focus frames. By way of example and for illustration, it is wellknown that out-of-focus images can be estimated as an in-focus image ofthe scene convolved with Gaussians. Gaussians with larger standarddeviations, when convolved with an in-focus scene image, result in amore blurred image, and conversely, convolving the in-focus scene imagewith Gaussians having smaller standard deviations would result in a lessblurred image. If we assume that pairwise similarity among said framesis proportional to (σ_(a)-σ_(b))², where σ is the standard deviation,the self-similarity matrix tabulates all pairwise similaritymeasurements. The frame corresponding to the Gaussian with the smalleststandard deviation will have the largest accumulated dissimilarities ascalculated by either shannon's entropy or a sum of squares method. Henceit will correspond to sharpest image.

[0165] The self-similarity matrix can be further manipulated in a numberof ways. A standard method of analyzing a symmetric matrix is to computeits eigenvalues. A special property of a symmetric matrix is that thesum of its eigenvalues equals the sum of its diagonal elements. Forinstance, a symmetric N by N matrix representing pairwise similaritieswill have N diagonal elements each having a value of 1.0. The sum of theeigenvalues for such a matrix, within numerical precision, is N.Eigenvalues represent roots of an N-degree polynomial represented by thesaid matrix.

[0166] When computed from frames acquired appropriately, derivedinformation from a self-similarity matrix may be used to distinguishvisual dynamic processes within a class. As is well-known in the art,the Hurst Parameter can be estimated for a time series ofself-similarity measurements. The Hurst Parameter can be used tocharacterize long-term dependencies. A self-similarity matrix and/orentropic indices can be analyzed to generate numeric evaluations of therepresented signal. Many variations on the above choices on how to useself-similarity can be used within the spirits of the invention toproduce similar results.

[0167] Standard statistical or matrix algebra methods can also be used.The following examples are illustrative only and do not limit the scopeof the invention described in the claims.

[0168] (a) Largest Eigenvalue of Self-Similarity Matrix

[0169] A self-similarity matrix representing a sequence of imagescontaining nearly identical scenes has an eigenvalue nearly equal to sumof its diagonal elements containing similarity match of an image toitself, 1.0 since a self-similarity matrix is a symmetric matrix. Asequence of images can be represented using said eigenvalue of saidself-similarity matrix. A plurality of said eigenvalues representing“signatures” resulting from applying a set of perturbations to a systemor set of similar systems can be used to rank said signatures with aconsistent measurement of the dynamics of the systems under eachperturbation.

[0170] (b) Periodicity of the Entropic Indices

[0171] Applying a self-similarity matrix to a frame sequence or imagesequence containing at least 2 whole periods of images representingperiodic motion such as that of a beating heart, and acquired withsufficient spatial resolution, produces entropic indices of the signalcontaining a dominant frequency at or near the periodicity of the imagedperiodic motion.

[0172] Self-Similarity as Motion Estimator

[0173] In some embodiments, self-similarity is estimated withoverlapping windows (602) and over a moving attention window (605).Specifically, frame geometry is sampled at S_(x), and S_(y). Definingthe top-left as the origin of the sampling window of 2_(x)S_(x), and2_(x)S_(y) in size, a self-similarity matrix is estimated for eachsampling window, as shown in FIG. 6. Sampling windows share 50 percentof their enclosed pixels with their neighboring sampling windows (602)(603) (604). In the illustrated embodiment, the entropic indices arecalculated and their standard deviation is used as an estimate ofrelative motion. Attention windows (605) are shifted forward one imageand self-similarity is estimated for the newly shifted attention window.

[0174] Exemplary and Watershed Frames

[0175] Exemplary frames are frames that represent epochs ofself-similarity in a frame sequence. Watershed frames are border framesthat separate exemplary frames. One aspect of the illustrated embodimentis the availability of the self-similarity matrix for deeper analysis oftemporal segmentation. A frame sequence that produces a flat integrationpath clearly describes an epoch best represented by a single exemplaryframe. Techniques exist in the art for identifying such frames andrelated results from self-similarity matrices. In some embodiments ofthe invention, exemplary and watershed frames can also be identifiedwhile frames are being acquired, thus allowing a novel set of choicesregarding further storage or processing of a given frame. In someembodiments, an accumulator image, accumulating the sum or the sum ofthe squares of the pixel values, with an appropriate depth, is createdat the beginning of an analysis step. The following operations can beperformed at the beginning of self-similarity estimation:

[0176] Note: if a new accumulator is needed, create one

r=correlation(ACM, i),  (11)

[0177] ACM is the accumulator image

if (abs(r−SM(i,i−1)<user-threshold)label_frame(existing exemplaryset)  (12)

else label_frame (candidate watershed frame)  (13)

[0178] In some embodiments, a set of user preferences can be used tospecify how many sequential watershed frames identify a watershed event.For instance, in certain dynamic processes, a user might be interestedin frames corresponding to events taking place in a very small number offrames. Such preferences could be established in units of time, framesor any other relevant metric.

[0179] Focus Deviation Detection

[0180] Auto-focus and measurement of image sharpness have been a focusof research and multiple methods exist for measuring image sharpness. Insome embodiments, the present invention detects deviation from focus ina continuous acquisition system. A self-similarity matrix of a framesequence containing in-focus and potential out-of-focus frames isanalyzed using (10) above. The choice of similarity measurement iscrucial in unsupervised classification of in-focus and out-of focusframes. As known in the art, the normalized correlation relationshipbetween two frames includes sharpness measurements of both images. Insome embodiments, continuous focus deviation detection is implementedusing a sliding measurement of self-similarity in a frame sequence.

[0181] Attentive Acquisition, Storage, and Control

[0182] The self-similarity matrix enables “selective” and/or “attentive”acquisition and/or storage of frames, i.e. “Attentive Capture.”Attentive capture enables the system to operate, e.g., acquire, analyze,and/or store frames at a rate that closely approximates the rate ofchange in information content in the scene.

[0183] Attentive capture is an adaptive control system that dynamicallyadjusts acquisition parameters and/or issues control directives toexternal devices. Acquisition parameters that can be adjusted include,but are not limited to, rate, exposure, aperture, focus, binning.Additionally, attentive capture, in combination with parameters definedin a preferred mode by a user, defined empirically and/or established bydefault, generates storage directives. A typical parameter that couldcontrol both acquisition and storage is the level of spatial frequencythat determines what constitutes an allowable level of change betweenframes. For example and without limiting the scope of the presentinvention, a biological assay may require the measurement of migrationor other motility statistics on cells. To facilitate such measurements,acquisition parameters could be controlled such that frame-to-framemovement of cells is minimized, and storage could be controlled suchthat images would be stored singly or over a period of time only when aperturbation in migration or other motility statistics is detected.

[0184] In a preferred mode, a user of attentive capture supplies thesystem with a few parameters describing the amount of motion or changeto attend to, an attention window size to assess self-similarity within,and an allowable deviation in self-similarity. In other modes, thesesettings could be default settings or determined empirically. During theinitialization steps as shown in FIG. 4A, a number of frames, N, equalto the size of the attention window specified, are loaded in to theimage buffer (301). In some embodiments, the number of frames is an oddnumber, though this could also be an even number. Said frames are markedas “must save” (433). One of skill in the art would recognize that anymarking scheme may be used at this step. In some embodiments, theparameter indicating the amount of motion to attend to can betransformed into a Gaussian low-pass filter representing a Gaussiankernel of standard deviation larger than said parameter (434). Theselected low-pass filter can be applied to all N images (441). Applyingsaid filter attenuates spatial frequencies higher than those prescribedby the filter. A self-similarity matrix can be computed, as outlined inFIG. 8, and the eigenvalues of said matrix can be calculated. Thelargest eigenvalue is normalized according to the teachings herein, andthis value represents an estimate of self-similarity for the frames inthe attention window.

[0185] In some embodiments, after the initialization step and after theacquisition of every frame, an assessment can be performed as to themarking of said image.

[0186] Co-Locomotion Module:

[0187] Objects in images and frames can be defined and located using awide range of their features. However, robust and accurate localizationof objects is most likely when using features that are the mostinvariant to predictable and unpredictable changes in the object, itsproximate surroundings, and the broader environment. If the objects arerigid bodies undergoing rigid transformation, one could assumeconservation of brightness, and use brightness as a defining feature tolocate objects. In many situations, however, this is not the case. Inbiological applications, for example, objects are deformable, undergomorphological transformations, and float in fluid or crawl along thesurface of culture dishes among other changes. In some embodiments, thepresent invention provides algorithms that detect such motion patternsto define objects.

[0188] Corpetti et al., “Dense Estimation of Fluid Flows,” IEEETransactions on Pattern Analysis and Machine Intelligence, 24,3:368-380(1998), incorporated herein by reference in its entirety, describe howthe deformable nature of fluid motion, the complexity of imagingprocesses, and the possible variations of temperature and pressure inmoving fluid all contribute to wide and unpredictable variations of theobserved brightness for a given element of fluid. In turn, this degreeof variation makes traditional feature detection for identifying objectsextremely difficult to apply to fluid systems. In contrast, imagesequences captured in accordance with the teachings herein reduce frameto frame transformation to translational change only (no rotation orshear), with linear changes in brightness regardless of the complexityof the underlying scene.

[0189] In some embodiments of the present invention, the co-locomotionmodule can be used to identify, assess, and track objects represented bya collection of adjacent pixels exhibiting motion in a similardirection. The co-locomotion module depends critically on an attentiveacquisition sub-system to capture frames at a rate that nearly oractually minimizes the derivative of the change from frame to frame. Animportant property of frames acquired at this borderline of minimalchange in information content is that frame-to-frame motion can bedescribed locally using only its translation component.

[0190] In some embodiments, motion vectors can be estimated using across-correlation of each small region of an “origin” image with acorresponding larger region in the image acquired next after the originimage. The larger region in the next-acquired image and the smallerregion in the origin image share a common center point. In someembodiments, by way of non-limiting example, larger or less squareregions such as can approximate a bounding region around arepresentation of a nematode worm in an image sequence, might beselected for cross-correlation instead of the general-purpose squareregions used in other embodiments. In still other embodiments, and alsoby way of non-limiting example, methods known in the art for definitionof rigid objects could be applied to define bounding boxes around eitherrigid or semi-rigid objects, and these bounding boxes could form thebasis of similar cross-correlations in accordance with the teachingsherein.

[0191] The co-locomotion module can estimate motion fields e.g., largerpatterns of motion in an image sequence derived through statisticalanalysis of collections of individual motion vectors. The co-locomotionmodule can also detect locomotive objects, track reference points withinand morphology of boundaries of said objects, compute aggregateper-frame velocity statistics, maintain state information on changingobjects over time, and maintain references to said object bounding imagedata across a frame sequence.

[0192] Motion Vectors

[0193] A motion vector is a vector in 3 dimensions x, y, and t (x-axis,y-axis, and time).

X=(x,y,t)^(T),  (14)

[0194] represents a vector in the spatiotemporal domain.

[0195] Given two successive frames, X_(i,j) is computed by matching“target,” a small square image window in frame t, with “search,” alarger square image window in frame t+1 that shares a common centerpoint with target. This operation is performed at the desired samplingrate for every pixel in frame t and t+1 that obeys boundary conditions.In some embodiments, target is 5 by 5 pixels, search is 9 by 9, samplingrate is 1. The center of the first target is at coordinates (searchwidth/2, search height/2) in frame t. The center of the first searchwindow is at coordinates (search width/2, search height/2) in frame t+1.

[0196] Target is correlated with search in the spatial domain using anormalized correlation, with a slight improvement in a preferred mode. Astandard normalized correlation of two images that contain constant andequal gray values is 0, implying no correlation. The preferred modifiedcorrelation is the square of the normalized correlation, which detectsthe zero case as a singularity in the underlying regression. Othermetrics that measure the sum or the square of absolute differencesbetween two images for use in the methods of the invention. Whereas astandard normalized correlation value is a normalized metric between−1.0 and 1.0, and whereas the square of a standard normalizedcorrelation yields values in the range of 0.0 to 1.0, the sum ofabsolute differences returns average gray value difference, which is notnormalized.

[0197] The correlation described herein can be performed in anexhaustive way. Target is matched with image data corresponding to amoving window of its size over search, resulting in a 5 by 5 correlationspace. The normalized correlation, which is squared in theaforementioned preferred mode, is: $\begin{matrix}{r = {{{Covariance}( {I_{1},I_{2}} )}\text{/}\sqrt{{Variance}( {I_{1}*{{Variance}( I_{2} )}} }}} & (15)\end{matrix}$

[0198] In some embodiments, having maintained an acquisition rate at orabove the rate of dominant motion or change in the scene, most if notall features of target have not moved outside of search. In areas offrame t and t+1 where the intensity function is nearly constant, thecorrelation space is flat and contains mostly singular correlationpoints. Furthermore, some correlation spaces contain poor correlationvalues. It is well known in the art that discarding the correctcorrelation spaces is accomplished by checking the highest correlationvalue with a figure of merit computed from local statistics. In theillustrated embodiment, we perform a spatial cross-correlation betweenframe t and t+1 to measure the degree of global motion between the twoframes, as shown in FIG. 7. Two low-pass versions of frame t and t+1 areproduced, G(t) and G(t+1). Then we perform the following correlations:

C _(t)=correlation(frame t, G(t+1))  (16)

C _(t+1)=correlation(frame t+1, G(t))  (17)

GME(global motion estimation)=sqrt(C _(t) *C _(t+1))  (18)

[0199] The correlation peak value is compared to GME. The correlationspace and its corresponding motion vector information is kept only ifthe correlation peak is above GME. Other embodiments may use residualsof a minimization process applied to the sum of the squared distancesbetween motion vector positions in t+1 and positions in t that haveundergone motion m. Yet other embodiments might use covariances of eachmotion vector displacement measurement. It is well known in the art thatmotion vector analysis can produce a motion vector for each point in animage. It is also known in the art that in practice, validation,correspondence and tracking of such motion vectors is ad-hoc. Theteachings herein demonstrate a systematic and generally applicable meansof validating, corresponding and tracking motion vectors.

[0200] If frame t and t+1 are exact copies of each other, thencorrelation peaks will all be at the center of the correlation space((2.5, 2.5) using the above values for search and target). A motionvector is a vector with the center of the correlation space as itsorigin and the peak as its second point. The location of the peak isestimated using a weighted moments approach. $\begin{matrix}{{x_{peak} = {\sum\limits_{ij}^{\quad}\quad {*{c( {i,j} )}\text{/}{\sum\limits_{i}^{\quad}{\sum\limits_{j}^{\quad}{c( {i,j} )}}}}}},} & (19) \\{{y_{peak} = {\sum\limits_{ij}^{\quad}\quad {j*{c( {i,j} )}\text{/}{\sum\limits_{i}^{\quad}{\sum\limits_{j}^{\quad}{c( {i,j} )}}}}}},} & (20)\end{matrix}$

[0201] Where x and y are the relative axis at i,j (0 through 4 using thedata above), and c is the correlation value at i,j.

[0202] With the above, then, X_(i,j)=(x_(peak), y_(peak), t).Immediately we can produce displacement and direction by computing theEuclidean distance from the origin to the peak and the arctangent ofpeak position.

[0203] Detecting and Characterizing Objects Based on Motion VectorCo-Locomotion

[0204] A locomotive object is an object defined by its motion. In someembodiments, locomotive patterns can be estimated using the totality ofmotion vectors for the first two frames. Detecting locomotive patternsinvolves assigning a label to each motion vector according to itsproximity to other motion vectors. Proximity can be described byEuclidean distance. For each motion vector we accumulate the square ofthe pairwise distance between it and every other vector. Motion vectorswith nearly identical total distance metrics receive similar labels.Objects defined based on this method can be further characterized basedon the identification of the major and minor axis of the object.

[0205] A collection of motion vectors with similar labels can in turn beused to define a bounding box around an object represented in the image.With the bounding box defined, further applications of a self-similarityfunction are possible, e.g., to characterize the relationship of thecontents of the bounding box in a given frame to the contents ofcorresponding bounding boxes in other frames. Other analyses of thebounding box and its contents are also possible, including, but notlimited to, analysis of pixel intensity or other values produced by thesensor and contained therein, and/or statistical analyses of these andother measurements across a collection of bounding boxes of the sameobject or multiple objects across space and/or time. Still otheranalyses include applying image segmentation based on raw intensity,texture, and/or frequency.

[0206] Higher-level statistics regarding motion and motion patterns canbe determined by applying standard statistical analyses to the totalityof motion information for every locomotive object over a given framesequence. Maheshwari and Lauffenburger, Deconstructing (andReconstructing) Cell Migration, Microscopy Research and Technique43:358-368 (1998), incorporated herein by reference in its entirety,suggests that individual cell path measurements can be used to predictcell population dispersion and also random motility. This outlines analgorithm for quantification of cell locomotion paths, using “center ofmass” as the canonical registration point on a cell. In general, anyrepeatable and accurate registration point will suffice. Using theapparatus and methods described herein, this algorithm can be applied tothe migration of each boundary point on a cell as well as a registrationpoint described above as center of motion. (21) S = ((Σs) / n) / dt(dt →) Translocation Speed (22) P = 2 dt / (Σφ²)/n Persistence Time (23) CI =(Σ(X · G))/L s

[0207] s is the Euclidean distance traveled in a time period,

[0208] n is the number of time points,

[0209] φ is the angle between successive displacements.

[0210] X is the displacement vector

[0211] G is the vector representation of the stimulus gradient

[0212] In some embodiments, at the conclusion of analysis for a givenframe, the most recently produced measurements can be applied to a setof application rules thus allowing the updating of the locomotive statefor each object.

[0213] Characterizing Dynamic Biological System and Biological Units.

[0214] In one aspect, the invention provides an apparatus, substantiallyas shown in FIG. 1, adapted for the characterization of biologicalunits. Here, the sensor(s) (102), which can take the form(s) of a CCD,CMOS, line-scanning camera, infrared camera or other sensor of interest,captures one or more images of the scene (101), which in this case is adynamic biological system. The dynamic biological system is contained ina suitable vessel, including but not limited to a slide, a flow chamber,a single-well Petri dish, a ninety-six well dish, or some othermulti-well dish suitable for the biological unit(s) under study. Amagnifying, amplifying or filtering device (109) can be used between thebiological unit and the sensor. Possible magnifying devices include butare not limited to a standard light microscope, a confocal microscope, astereo microscope, a macroscope and other wide field optics. Possiblefiltering devices include but are not limited to polarization, band-passand neutral density filters. The computing device (103) and storagedevice (104) are configured and operated as described above, as furthermodified as described herein in order to characterize the dynamicbiological system.

[0215] Characterizing Dynamic Biological Systems

[0216] In some embodiments of the invention, a dynamic system can be adynamic biological system. A “dynamic biological system” as referred toherein comprises one or more biological units. A “biological unit” asdescribed herein refers to an entity which is derived from, or can befound in, an organism. An “organism” as described herein refers to anyliving species and includes animals, plants, and bacteria or othermicroscopic organisms including protists and viruses. The biologicalunit can be living or dead, but is typically alive. Examples ofbiological units include cells, tissues, organs, unicellular organisms,and multicellular organisms. Also included are fragments of any ofthese, including cell fractions, e.g. membrane, nuclear or cytosolicfractions, and fragments or portions of organs, tissues, or organisms.Also included are subcellular objects, e.g., as described herein.Further examples include biological polymers (e.g. peptides,polypeptides, and/or nucleic acids), carbohydrates, lipids and ions. Thebiological unit can be either labeled or unlabeled. For example, a labelmight include an emitter, for example a fluorescent emitter, luminescentemitter or a radioemitter (e.g. alpha, beta or gamma emitter). A dynamicbiological system can be an independently selected combination of thesame or different biological units. Biological units can differgenetically, epigenetically, or phenotypically, as well as indevelopmental stage. Biological units can also be different by virtue ofmanipulation, e.g., treatments, e.g., exposure to one or more testcompounds. By way of non-limiting example, a dynamic biological systemcan be a single well on a multi-well plate comprising two or moredifferent cell types, two or more different organisms, or a combinationthereof. The biological unit can be incorporated into biological,nonliving or nonbiological material, e.g., cell surface proteins can beincorporated into a liposome. For example, a dynamic biological systemcan comprise neurons and glia, C. elegans and bacteria, or macrophagesand bacteria.

[0217] Any feature that can be detected by a sensor or combination ofsensors is referred herein as an “attribute”. For example, an attributecan be any feature of a biological unit that can be identified as analteration in the intensity of one or more pixels in an image. Oneexample of an attribute is the location of the plasma membrane which canbe detected as the difference in the intensity of light transmittedthrough the dish the cell inhabits and the intensity of the lighttransmitted through the cell. The attributes of biological units can bemonitored in response to the addition or removal of manipulations ortreatments. Manipulations can include altering temperature, viscosity,shear stress, cell density, oxygen tension, carbon dioxide tension,composition of media or surfaces contacted, electrical charge, oraddition of one or more other biological units of the same or differenttype. Such manipulations can be accomplished by methods commonly knownin the art. Treatments can include modulation (e.g. increasing ordecreasing absolutely, spatially, or temporally) of gene expression orprotein expression, and/or the addition or removal of a test compound,e.g. small molecules, nucleic acids, proteins, antibodies, sugars,lipids or complex natural or synthetic compounds. A test compound can bea compound with known or unknown biological function. The attributes ofbiological units can be used to characterize the effects of theabovementioned manipulations or treatments as well as to identify genesor proteins responsible for, or contributing to, these effects. Theattributes of biological units can also be used to characterize theinteraction between said biological unit and a second biological unit orother entity, e.g., a surface prosthetic device, a surgical implant, ora therapeutic device.

[0218] In some embodiments, the movement of subcellular objects can beevaluated using the illustrated method and/or apparatus. Examples ofsubcellular objects that can be analyzed in this manner include, but arenot limited to, proteins, nucleic acids, lipids, carbohydrates, ions,and/or multicomponent complexes containing any of the above. Furtherexamples of suitable subcellular objects include organelles, e.g.,mitochondria, Golgi apparatus, endoplasmic reticulum, chloroplast,endocytic vesicle, exocytic vesicle, vacuole, lysosome, nucleus. Themovement of subcellular objects can be from one compartment of the cellto another, or can be contained within a single compartment. Forexample, a protein localized at the plasma membrane can traffic to thecytoplasm or nucleus, or can simple move from one region of the plasmamembrane to another.

[0219] In some embodiments the method and/or apparatus described hereinis used to monitor the state of the DNA in a dividing cell in order tocharacterize cell division. Cells described herein as appropriate foruse in the analysis of cell division are also suitable for thisembodiment, as are the experimental conditions described above. The DNAcan be visualized by means of a fluorescent, vital dye, e.g., Hoechst33342 or SYTO dyes (available from Molecular Probes), or through the useof polarization microscopy, as well as other means. In the case wherethe DNA is visualized via fluorescence, the illustrated apparatus mustbe modified to include appropriate excitation and emission filters. Asthe cells enter M phase, the DNA condenses and the otherwise diffusepattern of nuclear fluorescence becomes localized first into punctatestructures and then into discernable chromosomes. Chromosomes can beidentified and tracked based on the colocomotion of motion vectors. Thechromosomes then align at the center of the cell. Once at the center ofthe cell the chromosomes overlap visually and appear to be one largemass of DNA. Once the chromosomes begin to separate they can again bedetected using motion vectors and tracked while they move towards thetwo poles of the cell and segregate into the two daughter cells. Basedon the pattern of appearance, coalescence into one structure andseparation into individual structures, the state of the DNA throughoutmitosis can be assessed and used to evaluate the impact of manipulationsand treatments on this complex process. This information can be usedsubstantially as described above for mitosis.

[0220] Screening is the process of evaluating a plurality ofmanipulations, treatments or test compounds for their ability tomodulate an attribute, or some other parameter of interest, e.g.,affinity, between two similar or different biological units or between abiological unit and a treatment or test compound, interaction betweentwo similar or different biological units or between a biological unitand a treatment or test compound in a computer based simulation or model(also known as rational drug design). The attributes of biological unitscan be used as a primary screen, e.g., to identify manipulations ortreatments that are capable of modulating specific cellular attributesfrom a larger set of manipulations and treatments. Such a screen is saidto be “high-throughput” if the number of manipulations and treatments isgreater than 1,000. Attributes of biological units can also be used as asecondary screen, e.g., to further assess the activity of theaforementioned manipulations or treatments after their identification byanother means, including, but not limited to, a previous screen.Furthermore, attributes of biological units can also be used assess therelationship between properties of treatments or a series of treatments,a process also known as determining structure-activity relationships. Inthis case, two or more treatments that share a similar property can beevaluated using the methods of the invention and can be the relationshipbetween the similar property and an effect of treatment on an attributeevaluated. Treatments identified in any of the abovementioned mannerscan be further evaluated by deriving a series of next generationtreatments, e.g. a new treatment that has been modified in one of moreways from the first treatment identified, which can then be evaluatedusing a similar or different method or apparatus.

[0221] A manipulation or treatment identified as a modulator of anattribute of a biological unit can function in the extracellular orintracellular space, e.g., plasma membrane, cytoplasm, mitochondria,Golgi apparatus, endoplasmic reticulum, chloroplast, lysosome, nucleusor other organelle. Based on these findings, manipulations or treatmentscan be developed as therapeutics with activity against diseasescharacterized by alterations in the attributes under study, ordiagnostic tests for said diseases.

[0222] After the identification of manipulations or treatments withdesired effects, the mechanism of action of these manipulations ortreatments can be explored. One method for exploring the mechanism ofaction of a test compound or combination of test compounds is toidentify polypeptides, nucleic acids, carbohyrates, lipids or ions thatit interact with it.

[0223] This interaction can be identified using affinity-basedpurification as known in the art. This interaction can also be assessedusing the technique commonly known as a “drug western” in which atreatment is labeled with a fluorophore or radioemitter and is used toprobe an expression library. Alternatively, this interaction can beassessed using phage or cell display methods, where the interaction ofphages or other cells expressing a library of proteins or polypeptidesis used to identify proteins that interact with the treatment understudy.

[0224] In addition to screening for manipulations and treatments thateffect attributes of dynamic biological systems and biological units,the method and apparatus described herein can also be used to evaluatethe activity of a gene. Gene activity can be modulated either at thelevel of the DNA (e.g., by targeted mutagenesis, or random mutagenesis),mRNA (e.g., by using RNAi, antisense RNA, or a ribozyme) or protein(e.g., by using a test compound, antibody, or other protein thatinteracts with the protein product of the gene under study). Geneactivity can also be modulated by manipulating the biological unit.Furthermore, the activity of multiple genes can be modulated at the sametime. Attributes of control cells and of cells where gene activity hasbeen modulated can be compared and the activity of the gene under studyis thus evaluated.

[0225] Examples of cellular attributes that can be evaluated using theseanalytical methods include, but are not limited to, cell morphology andmorphological change (e.g., contraction, spreading, differentiation,phagocytosis, pinocytosis, exocytosis, polarization), cell division(e.g., mitisos, meiosis), cell motility, cell death (e.g., apoptosis ornecrosis), and cell adherence. Examples of subcellular attributes thatcan be evaluated using these analytical methods include, but are notlimited to, the expression, localization, or translocation of proteins,nucleic acids, lipids, carbohydrates, ions, multicomponent complexescontaining any of the above. Further examples of subcellular attributesinclude the localization and number of organells, e.g., mitochondria,Golgi apparatus, endoplasmic reticulum, chloroplast, endocytic vesicle,exocytic vessicle, vacuole, lysosome, nucleus. Examples of organismalattributes that can be evaluated using these analytical methods include,but are not limited to, organismal motility, organismal morphology andmorphologic change, organismal reproduction, organismal development, andthe movement or shape change of individual tissues or organs within anorganism. Attributes can be monitored through inspection with any one orcombination of the sensors described above and are not limited toattributes visible via detection of either visible or fluorescent light.

[0226] A range of attributes that can be analyzed using the methods andapparatus described herein are detailed in FIG. 10. Specific embodimentsof the analysis of the attributes of biological units are discussedbelow.

[0227] In some embodiments, the method and/or apparatus described hereincan be used to characterize cell morphology. Morphology is important asa marker of many general cellular properties including, but not limitedto, viability, mitosis, migration, adhesion, phagocytosis,differentiation and death. Morphologic change is also a feature ofspecific cellular events including, but not limited to, smooth andcardiac muscle contraction, platelet activation, neurite outgrowth, axongrowth cone guidance, oncogenic transformation, white blood cellmigration, white blood cell phagocytosis, and cancer cell migration. Anautomated means for analyzing morphology and morphologic change hasbroad applications in drug discovery and basic science research.

[0228] One example, not meant to be limiting, of morphologic change, iscell spreading, e.g., platelet spreading. Platelets are one of thecellular components involved in blood clot formation and morphologicchanges are widely considered to be important markers of the plateletactivation process. During the spreading process, platelets transitionfrom a rounded morphology to a flat morphology. Platelets can be imagedduring the spreading process using the illustrated embodiment with asuitable magnification device, for example a microscope. Mammalianplatelets can be purified from whole blood using a number ofwell-established methods. Isolated platelets can then be placed in asuitable vessel and allowed to adhere to the surface. It is widely knownthat the surface properties of the vessel, i.e. the material itself aswell as any coating or treatment, are important in determining whetheror not the platelets will adhere and spread. It is also widely knownthat substances including ADP, fibrin and others, can be added to theplatelet mixture to further activate the platelets and promote adherenceand spreading. Thus, the invention includes methods of evaluating theeffect on cell spreading, e.g., platelet spreading of manipulation ofsurface properties of a vessel containing the cells, and/or the additionof test compounds, including but not limited to ADP, fibrin, and thelike.

[0229] Images of platelets are difficult to analyze because plateletsare extremely small compared to other cells. Before spreading, roundplatelets generally measure between 1 and 5 microns in diameter. Oncespread, platelets generally measure between 3 and 10 microns in diameterand between 1 and 3 microns in height. These dimensions result intransmitted light images that are low contrast using generally appliedoptical techniques such as phase contrast or differential interferencecontrast. As a result, it is difficult to perform detailed analysis ofspreading based on morphological analysis using traditionalintensity-based thresholding techniques without substantial humaninvolvement.

[0230] In some embodiments, self-similarity can be used to analyzeplatelet spreading, thus eliminating the need to identify each plateletindividually. Platelets can be purified from mammalian blood and washedusing known centrifugation techniques. Purified, washed platelets areplaced on a permissive substrate and/or activated with a suitableactivator. Permissive substrates include, but are not limited to,plastic and glass, either untreated or treated with proteins, chemicalsor nucleic acids. Other permissive substrates include biologic surfacessuch as vascular endothelium (e.g. a plastic tissue culture dish surfacecoated with a monolayer of adherent human umbilical vein endothelialcells), or disrupted or modified vascular endothelium in intact orisolated blood vessels. Activators include, but are not limited to, vonWillebrand factor, collagen, fibrinogen, fibrin, as well as proteolyticfragments of any of the above, ADP, serotonin, thromboxane A2. Shearstress is also an activator of platelets. During this process theplatelets can be imaged using a suitable magnifying device and sensor,and the images are made available to a computing device for analysis.

[0231] In some embodiments, the computational approach described abovefor the calculation of self-similarity, either for the entire scene orfor each individual platelet, can be applied to a sequence of imagesthat depict platelets either spreading or not spreading in response toone or more manipulations or treatments. Because their morphology ischanged by the spreading process, platelets that are spreading will havea lower degree of self-similarity than platelets that are not spreading.This morphologic information can be used as a surrogate for spreadingand provides information about the impact of each manipulation ortreatment under study. Thus, without specifically determining where theplatelets are in each frame, or what their morphology is on anindividual basis, self-similarity can be used to analyze the spreadingprocess.

[0232] In addition to platelet spreading, self-similarity can be used toanalyze any cell or organism that is changing shape, including, but notlimited to skeletal, cardiac and smooth muscle under conditions thatstimulate contraction (i.e. electrical stimulation, adrenergicstimulation, as well as other suitable physical and chemical stimuli).In some embodiments, the analysis of cell shape change described abovecan be employed to screen for manipulations and treatments that could beused to treat diseases of platelet activation, including but not limitedto, deep venous thrombosis, peripheral artery occlusion, myocardialinfarction, embolic stroke and pulmonary embolism, as well as disease ofaltered muscle contraction, including, but not limited to, hypertension,heart failure and chronic skeletal muscle contractures.

[0233] In another embodiment, the methods and/or apparatus describedherein can be used to analyze cell motility. Cell motility is central toa wide range of normal and pathologic processes including, but notlimited to, embryonic development, inflammation, tumor invasion andwound healing. Cell motility is highly variable from cell to cell andbetween cell types. In order to identify and analyze moving cells,existing image processing tools require either substantial humanintervention or cell lines that have been genetically engineered to befluorescent, a property that aids in image segmentation. An automatedapparatus for analyzing cell migration, such as that described herein,is required to screen large numbers of manipulations or treatments inorder to identify modulators of cell migration that may have broadtherapeutic applications in a number of diseases. Autoimmune andinflammatory diseases are examples of diseases associated with changesin cell motility. Specific examples of said disease include, but are notlimited to, rheumatoid arthritis, systemic lupus erythematosis,myesthenia gravis, ankylosing spondylitis, psoriasis, psoriaticarthritis, asthma, diabetes, atherosclerosis and transplant rejection.In addition to inflammation and autoimmune disease, cell motility isimportant for cancer, both solid tumors and hematologic malignancies,including carcinomas, sarcomas, lymphomas, leukemias and teratomas. Cellmotility is also important for neuron and axon growth cone migration,pollen tube growth, and pathogen motility.

[0234] In some embodiments the methods and/or apparatus as describedherein can be used to characterize white blood cell motility. This canbe motility of primary white blood cells and/or an immortalized whiteblood cell line. Primary white blood cells can include lymphocytes,monocytes/macrophages, neutrophils, eosiniphils, and basophils, and canbe prepared from mammalian blood. Immortalized white blood cell linescan include Jurkat, A20, AD10, Peer, L1.2, HL-60, PLB-985, THP-1, U-937,MonoMac6, K-562, AML14.3D10 (all of which are available from ATCC), aswell as other cell lines characterized to be either normal or pathologiccells of the lymphoid and myeloid lineages.

[0235] In one example the white blood cell line HL-60, is grown in aflask, dish, multi-well dish or other suitable culture dish. The whiteblood cell line is induced to differentiate by one of a number of wellcharacterized means, including treatment with DMSO or retinoic acid.Once differentiated, the white blood cell line is stimulated with anagonist of cell motility. The agonist can be applied to the entirepopulation uniformly, or can be released from a point source in order tocreate a gradient of agonist. Agonists of cell motility includecytokines, chemokines, other products of inflammation, components ofcomplement, other small molecules, ions and lipids. In this embodimentthe preferred agonist of cell motility is a chemokine. Examples ofchemokines include, but are not limited to, IL-8, GCP-2, Gro alpha, Grobeta, Gro gamma, ENA-78, PBP, MIG, IP-10, I-TAC, SDF-1 (PBSF), BLC(BCA-1), MIP-1alpha, MIP-1beta, RANTES, HCC-1, -2, -3, and -4, MCP-1,-2, -3, and -4, eotaxin-1, eotaxin-2, TARC, MDC, MIP-3 alpha (LARC),MIP-3beta (ELC), 6Ckine (LC), I-309, TECK, lymphotactin, fractalkine(neurotactin), TCA-4, Exodus-2, Exodus-3 and CKbeta-11. Agoniststimulation promotes cell adherence to contacted surfaces as well ascell motility. After agonist stimulation, cells are allowed to adherefor 1 hour and non-adherent cells are washed off. Images of the whiteblood cells are acquired (201) using a sensor (102) with an appropriatemagnifying device (109) and acquired images are analyzed (203) using adata processing device (103), as described below. After analysis, dataand frames are stored using a suitable storage device (104) whichenables reporting (203) of the data.

[0236] In this embodiment, analysis (203) is a multi-component processthat can include one or more approaches. Self-similarity between theimages in the sequence is calculated. Self-similarity can be used toensure that a minimal number of frames are acquired without missingimportant events by dynamically modulating the frame rate of the camerabased on this measurement, described above as attentive acquisition.Alternatively, self-similarity between the images can also be used as ameans for obtaining a global representation of the cell migrationprocess (i.e. as a motion estimator) in order to establish a signatureof the cell migration under the specific experimental conditionsemployed. Alternatively, self-similarity can also be used to identifyexemplary and watershed frames as landmarks in the video sequence thatmark the location of events of interest (equations 11-13).Alternatively, self-similarity can be used to identify frames where thefocus has deviated from the established focal plane. Based on thisidentification, frames, e.g., artifactual out of focus frames, can bemarked, removed from further analysis, and/or discarded.

[0237] Using techniques described herein, motion vectors can becalculated and a motion field created. Each object in the imagesequence, cell or otherwise, can be localized based on the co-locomotionof motion vectors. Motion vectors can be used to calculate the aggregatevelocity of all the white blood cells, as well as the velocity of eachcell. Cell localization based on motion vectors also allows theestablishment of temporal reference points, including, but not limitedto the center of motion. By tracking temporal reference points,velocity, direction and other spatial metrics can also be calculated foreach cell in every frame. In the case where the agonist of cell motilityis released from a point source, directional movement towards thatsource is typically expected. By way of non limiting example, theaggregate direction and speed of an object can be calculated based onthe sum of the motion vectors associated with it object. In addition,the direction and speed of the center of projections can be used toevaluate object motility.

[0238] Once a cell has been identified and characterized using one ormore of the abovementioned parameters, it can be assigned to categoriessuch as: not-moving, moving without directionality, moving withdirectionality, dividing, etc. By way of non-limiting example, anot-moving cell can be defined as one for which the magnitude of itsaggregate motion vector is zero for the relevant temporal window. A cellmoving without directionality can be defined as one for which thesummation of its motion vectors is zero, or close to zero, during therelevant temporal window. A cell moving with directionality can bedefined as one for which the summation of its motion vectors is non-zeroduring the relevant temporal window. A dividing cell can be defined asone for which, during the relevant temporal window, one object ofinterest with one center of motion is transformed into two separateobjects of interest with separate centers of motion. Thesecategorizations can be used to further characterize cell motility or theimpact of a manipulation or treatment on cell motility.

[0239] Self-similarity, as well as the parameters described above, oranother suitable parameter or set or parameters, can also be used toregulate the storage of images of migrating cells in order to furtherreduce the number of frames stored for each experiment. For example, theproportion of frames stored for any experiment can be dynamicallycontrolled based on the degree of similarity that a single image has tothe larger sequence of images being acquired in that experiment.Alternatively, the proportion of frames stored could be controlled basedon some combination of the speed, direction, persistence, or othersuitable parameter being measured in every frame. By storing framesbased on self-similarity or other parameters, the number of storedframes is decreased and the amount of memory required for eachexperiment is decreased. This process is carried out by the selectionmodule (211) and results in the conversion of attached data into XMLdata (212) and the encoding (213) of frames for storage (214) based on aseries of user preferences.

[0240] Other examples of cells whose movement can be analyzed in themanner described herein include epithelial cells, mesenchymal cells,cells from the nervous system, muscle cells, hematopoietic cells, germcells (e.g. sperm), bacteria and other single-cell organisms. In eachcase, cells are grown in a suitable culture device under conditionsspecific to that cell type. Cell movement can then be analyzed asdescribed herein, e.g., as described for white cells. In all cases,these forms of characterization can be used to establish the impact of amanipulation or treatment on cell migration, e.g., for the purpose ofcharacterizing each substance or treatment and deciding which substanceor treatment may have the potential to be therapeutically ordiagnostically relevant.

[0241] In another embodiment, the methods and/or apparatus describedherein can be used to analyze cell division (e.g. mitosis or meiosis).Cell division is a complex and essential process for all living cellsand organisms. The cell division cycle is generally considered toconsists of four phases, G1, S, G2, and M. During G1, S and G2 mostcells retain the morphology characteristic of that cell type. During Mmost cells round-up to assume an approximately spherical morphology,then segregate the chromosomes to two poles established within thesphere, and then the sphere is cleaved at a plane between those twopoles.

[0242] Dividing cells can be studied by the methods described herein,including dividing mammalian cells, as well as other animal cells,yeast, bacteria and unicellular organisms. In some embodiments anadherent cancer cell line is studied, e.g., a cancer cell line includingbut not limited to, the cell lines MCF-7, BCap37, MDA-MB-231, BT-549,Hs578T, HT-29, PC-3, DU-145, KB, HeLa, MES-SA, NIH-3T3, U-87, U251,A549-T12, A549-T24 (all available from ATCC). Non-adherent cell linescan also be studied using the methods and apparatus described herein,although the change in morphology from flat to round does not occur.Otherwise, non-adherent cells can be analyzed as described herein foradherent cells. In order to increase the number of dividing cellsobserved in any time period, cells can be synchronized using knownmethods such as thymidine blockade, and/or serum starvation. Inaddition, cells can also be induced to divide using growth factors,re-administration of serum after starvation, radiation, or other knowntechniques.

[0243] Self-similarity changes relatively more at the start and end ofcell division because these are periods of accelerated change from onemorphology to another. As described above, at the start of cell divisionthe cell changes from flat to spherical morphology and at the end thetwo resulting spheres transition to flat morphologies again. Thesedramatic periods of decreased self-similarity can be used as markers toidentify the presence of dividing cells and to measure the length oftime they spend in cell division.

[0244] In a further embodiment dividing cells can be identified usingthe pattern of motion vectors for each cell. In this method, the patternof motion vectors for each cell is used to identify the cleavage plane.In a stationary spherical cell the center of motion can be used toestablish the center of the cell. During cell division the plasmamembrane of the cell is drawn inwards along a plane that intersects, ornearly intersects the center of the cell that is generally perpendicularto the axis of view of the sensor. As a result, a collection of motionvectors will exhibit a high degree of symmetry, largely pointingcentripetally along a single axis. As the cell continues throughdivision these centripetally oriented motion vectors graduallyreorganize their orientation to identify two centers of motion thatcorrespond to the future center of each future daughter cell. Based onthis signature, cells can be identified as dividing.

[0245] It is generally appreciated that uncontrolled or improperlycontrolled cell division contributes to the development of cancer, otherdisease that involve excessive cellular proliferation, as well as otherdiseases and malformations. As a result, cell division is the subject ofa tremendous amount of research and pharmaceutical development. Asystem, such as the one described herein, can be used in a broad rangeof applications that include, but are not limited to, testingmanipulations or treatments for their potential role in mammalian cellsas anti-proliferative agents and anti-cancer agents, as well as indiagnostic evaluation of cancer cells.

[0246] In another embodiment, the methods and/or apparatus as describedherein can be used to analyze programmed cell death, also known asapoptosis. Apoptosis is central to the regulation of cell number inorganismal development and throughout the life of an organism, and isimplicated in a wide range of diseases from cancer to autoimmunity.Apoptosis results in a characteristic series of morphological changes incells that have been well characterized, including, but not limited to,the arrest of cell motility and the onset of membrane blebbing.

[0247] In some embodiments a cell line, e.g., a cancer cell line isstudied and can include any of the adherent cell lines described hereinfor cell division, and can also include a number of cell lines that growin suspension, including, but not limited to HL-60, MOLT-4, and THP-1(all available from ATCC), as well as other cell lines derived fromleukemias or lymphomas. The arrest of cell motility is determined basedon the assignment of temporal reference points, as described herein. Forexample, a cell can be said to have arrested motility if the center ofmotion moves less than 10% of either dimension of the cell in a periodof 10 minutes. Membrane blebbing associated with apoptosis, or othercellular processes, is detected based on the clustering motion vectorsat the surface of an otherwise non-motile cell. Blebs result in rapidlychanging microdomains at the surface of cells that have a characteristicsize and time course. The presence of rapidly changing domains of motionvectors, e.g., domains that contain 3 or more motion vectors that lastfor 10 minutes or less, at the boundary of the cell without acorresponding change in the center of motion is indicative of apoptosis.

[0248] The method of evaluating apoptosis described herein can be usedto automate screening for manipulations or treatments that eitherpromote or prevent apoptosis. For example, such an embodiment can beused to identify manipulations or treatments that promotechemotherapy-induced apoptosis or radiation-induced apoptosis of cancercells but not normal cells, or that selectively kill specific subsets ofT- or B-cells. This embodiment can also be used to identifymanipulations or treatments that prevent apoptosis in response toischemia and reperfusion, e.g., in stroke and myocardial infarction.This method could further be used as a diagnostic test for the frequencyof apoptosis or the frequency of apoptosis in response to a manipulationor treatment. This information can be used for the diagnosis of adisease or the choice of a therapeutic agent based on its ability toinduce apoptosis.

[0249] In another embodiment, the method and/or apparatus describedherein is used to analyze cell adherence. Cell adherence is a dynamicprocess that depends both on the substrate and the cell, and is highlyregulated at both levels. Cell adherence is important in normaldevelopment and physiology as well as in pathologic conditions such as,but not limited to, tumor invasion and metastasis, inflammation, axonguidance, atherosclerosis and angiogenesis. Cell adherence can bemeasured in a number of ways, including, but not limited to, placingcells in a culture dish for a defined period of time and then washingaway any non-adherent cells and placing cells on a slanted surface andobserving the number of cells that are stationary or rolling.

[0250] In another embodiment, cells are passed over a surface by virtueof their suspension in liquid in an apparatus commonly referred to as aflow chamber. The cells can include, but are not limited to, white bloodcells, platelets and cancer cells. In each case both primary andimmortalized cell lines representing these three categories of cells aresuitable for analysis using the illustrated embodiment. Some examples ofappropriate immortalized cell lines include HL-60, THP-1, U937, and K562(all available from ATCC). Generally, the cells under investigation canbe any cell type capable of adhesion, and the surface can be any solidor semi-solid material that supports the adherence of the cell typechosen for analysis.

[0251] In one example, primary human monocytes can be purified fromwhole blood using Ficoll-Hypaque density-gradient centrifugationfollowed by magnetic bead purification and can be passed over thesurface of a flow chamber consisting of human umbilical vein endothelialcells (HuVEC) growing on the bottom of the chamber. These endothelialcells can be engineered to express specific adhesive receptors,including but not limited to, E-selectin, P-selectin, ICAM-1, VCAM-1, topromote adhesion and rolling. As cells pass over the endothelial cellsurface a proportion of the flowing cells adhere and roll on the surfaceof the endothelial cells. Analysis of cell rolling can be performedusing a number of different features of the illustrated invention.Useful approaches described herein include tracking the cellsindividually, analyzing their movement, and characterizing the wholescene by virtue of self-similarity. In the first embodiment, cells canbe localized using motion vectors, and tracked by virtue of theassignment of temporal reference points. In this case, the center ofmotion is particularly well suited to this analysis by virtue of thecell's predictably round shape. Based on these temporal referencepoints, velocity can be calculated for both the flowing cells and therolling cells, and the proportion of rolling cells can be determinedbased on their slower rate of rolling. Additionally, their rate ofrolling and duration of rolling can be calculated based on theirtransition from a fast-moving to a slow-moving state.

[0252] In a further embodiment, whole characterization of the sceneusing self-similarity is employed to detect periods of difference withinthe experiment. If no cells adhere, the frames relate to each other in aconsistent manner that is determined by the frequency of flowing cellspassing in front of the sensor. As long as this frequency isapproximately constant, the self-similarity should remain approximatelyconstant. Whenever a cell adheres to the surface and begins to roll itwill produce a decrease in self-similarity which can be used as asurrogate for an analysis of the cell itself. In this way,self-similarity can also be used as a motion estimator and can thus beused as an efficient and robust measure of rolling without specificassignment of features to any given cell. This approach is particularlyvaluable when large numbers of cells are passed in front of the sensorin each experiment.

[0253] The methods described herein for analyzing cell adhesion can beapplied to the discovery of manipulations or treatments that modify celladhesion. Such manipulations or treatments would be useful in treatingor preventing a wide range of conditions including, but not limited to,cancer (by preventing tumor metastasis), inflammation (by preventingleukocyte homing to sites of inflammation) and thrombosis (by alteringplatelet adhesion and rolling). The illustrated embodiment can also beused as a diagnostic test of conditions charaterized by decreased celladhesion, including von Willebrand disease, Bemard-Soulier syndrome,Glanzmann thrombasthenia, Leukocyte Adhesion Deficiency I, and LeukocyteAdhesion Deficiency II.

[0254] In another embodiment, the method and/or apparatus as describedherein can be used to analyze the movement of a unicellular ormulticellular organism. This organism can be chosen from a list thatincludes, but is not limited to, Listeria species, Shigella species, E.coli, Dictyostelium, C. elegans, D. melanogaster, D. rerio as well asother organisms. In multicellular organisms, movement is a complexprocess that requires the integration of multiple cell types within theorganism to produce a coordinated behavior. As such, movement can beused to study the functioning of each of the cellular componentsinvolved as well as their integration into a properly functioningsystem.

[0255] In embodiment, C. elegans (referred to below as “worm”) motilitycan be analyzed. In this embodiment the worm can be a wild-type worm ora worm harboring a genetic mutation or other alteration in gene orprotein expression. In order to analyze worm movement, motioncharacteristics can be calculated using either a simple aggregatecenter-of-motion scheme, or using the medial-axis method calculatedbased on opposing motion vectors. The medial-axis method identifies thebody of a worm by the collection of medial points between all paired,opposing motion vectors.

[0256] The methods described herein can be used to screen formanipulations or treatments that affect each component of the worm thatcan be involved in movement, including but not limited to theneurotransmitter systems employed. For example, an automated analysis ofworm movement is used to identify treatments that modulate locomotorybehavior controlled by the serotonin neurotransmitter system in aneffort to identify substances with a selected effect on this complexsystem that has been implicated in human mood and clinical depression.

[0257] In another embodiment, the methods and/or apparatus as describedherein can be used to evaluate organismal development. Organismaldevelopment is that period in an organism's life when it has yet toattain its mature or adult form, e.g. while in the egg, uterus, or otherreproductive organ, or while outside the reproductive organ butconsidered to still be in the process of attaining a mature or adultform. Examples of organisms whose development can be analyzed using theillustrated method and/or appatatus include C. elegans, D. rerio, X.laevis, D. melanogaster, chicken, domesticated cow, M. musculus, and H.sapiens. Organismal development generally occurs over a period of hoursto days or months, and as such is not readily amenable to continuoushuman observation. As a result, an automated system for the analysis ofembryonic development is valuable to a range of activities from in vitrofertilization to the study of embryology and the evaluation ofmanipulations and treatments for their effect on events duringorganismal development.

[0258] In one embodiment, a human embryo is observed after in vitrofertilization. After fertilization, the embryo is maintained undercontrolled media conditions widely known in the art and can be monitoredusing a microscope and a suitable sensor, e.g., a CCD camera, enclosedin climate controlled chamber (e.g., constant temperature, CO2, O2 andhumidity). One embryo is placed in each well of the culture dish. Theembryos are monitored constantly for 3 days after fertilization. Celldivision events are detected as described for mitosis above, usingeither of the two methods, however using changes in self-similarity ispreferred. Over the 3 days of monitoring, the timing of each celldivision is precisely recorded. It is expected that the embryo willundergo three mitoses, thus reaching the eight-cell stage. Based on thetiming and number of mitoses, as well as other features such as cellsymmetry and cell morphology, embryos can be chosen for implantation, orfor further incubation until day five when they will have reached theblastocyst stage.

[0259] In another embodiment, the methods and apparatus described hereincan be used to assess the behavior of an organ within an organism. Theappropriate coordination of heart rate, rhythm and contractility arecritical to the survival of organisms with a cardiovascular system, anddefects that alter these parameters result in arrhythmias and/or heartfailure and reduced survival. Heart rate, rhythm and contractility canbe studied by visualizing the heart directly or by monitoring itsactivity based on hemodynamic or electrical sensors. In many developingorganisms, as well as some adult organisms, it is possible to analyzethe movement or activity of individual organs due to the transparency ofthe embryo or organism. Additionally, the heart and other organs can bemade visible through the use of x-rays or other non-invasive imagingmodalities, such as CT or MRI, with or without the addition of contrastmedia, depending on the organ and imaging modality. Therefore, imagingis an effective means for studying heart rate and rhythm in any organismwhere the movement of the heart can be visualized and an appropriatesystem is available for automated analysis.

[0260] In one embodiment, heart rate, rhythm and contractility areanalyzed in D. rerio, also referred to herein as “zebrafish,” embryos orlarvae using the illustrated invention. Poorly pigmented mutants(Albino, Brass, Transparent) are preferred due to their greatertransparency. Zebrafish used in this embodiment can also carry inducedor spontaneous mutations that are either known or unknown to theinvestigator. Embryos and larvae can be studied at, for example, threeto six days post fertilization. Animals are placed in a suitable vesseland may be anesthetized with phosphate-buffered tricaine and/orimmobilized in low melting point agarose. Animals are imaged over amultiplicity of cardiac cycles and subject to analysis. The heart isidentified based on the periodicity, e.g., the magnitude of theperiodicity of motion vectors associated with it. The rate and rhythm ofthe cardiac cycle is identified and analyzed using the periodicity ofits self-similarity during successive heart beats. Its size can becalculated based on geometric measurements, e.g., major and minor axis,obtained at periods in the cardiac cycle known to correspond to diastoleand systole. Based on these dimensions, contractility can be assessed.

[0261] The methods and apparatus described herein can be used to analyzezebrafish that are part of or have been generated by a systematicmutagenesis screen or a screen for manipulations or treatments thatalter cardiovascular function. More generally, this embodiment can beused to analyze the rhythmic contraction of any organ or tissue that canbe visualized or acquired using a suitable sensor and rendered into aspatiotemporal signal. Manipulations or treatments discovered based ontheir ability to modulate smooth, cardiac or skeletal muscle functionare potential therapeutics for medical diseases or conditions whichresult in or from altered muscle contraction, including, but not limitedto hypertension, heart failure, inflammatory bowel disease, irritablebowel syndrome, skeletal muscle contractures, uterine contractionsduring labor, and hyperactive bladder syndrome.

[0262] In another embodiment, the methods and/or apparatus describedherein can be used to evaluate the interaction of a biological unit witha surface. Interaction of biological units with surfaces is a complexand essential process that is central to an understanding of manyphysiological processes, such as a cell's interaction other cells,tissues and organs (e.g. bone or transplanted tissues and organs), andartificial surfaces such as prosthetics and implants. For example,platelets adhere to glass, plastic, or other manufactured surfaces andthis interaction can be used as a surrogate for their interaction withendothelium or clot. Other examples of interaction between and amongbiological units and surfaces include, but are not limited to,fibroblasts interacting with the extracellular matrix, and cancer cellsadhering to endothelial cells in the process of metastasis, andlymphocyte synapsis with antigen presenting cells during immunereactions. Still other examples of interaction between biological unitsand manufactured surfaces include kidney cells adhering to an artificialscaffold and fibroblasts adhering to medical devices such as orthopedicimplants, artificial heart valves, and cardiac defibrillators.

[0263] In some embodiments, the surface with which the biologic unitsare interacting is uniform. Examples of uniform surfaces includeinorganic substances such as steel, titanium, aluminum, ceramic, glass,and quartz, as well as organic substances such as plastic andfiberglass. In other embodiments the surface is variable, either interms of gross surface roughness, or in terms of engineered variabilityvia mechanical etching, plasma etching, or lithography. In still otherembodiments, the surface is comprised of pores, openings, concavities,convexities, smooth areas and rough areas. Examples of such surfacesinclude micro-machined crystalline silicon, as well as nanotubes andpatterned polymers. In still other embodiments, the surface variabilitycomprises changes in composition. An example of compositional changeincludes variability based on composite “sandwiches” made from carbonfiber and epoxy. In still another embodiment, the surface variabilitycomprises change in charge. An example of a charged surface includes atwo-dimensional array of impedance electrode elements, or atwo-dimensional array of capacitance electrode elements. In still otherembodiments, surface variability could comprise the presence or absenceof a treatment (e.g., a test compound), either in uniform concentrationor in a gradient. Examples of test compounds include agonists such ascytokines, chemokines, other products of inflammation, components ofcompliment, small molecule, ions and lipids.

[0264] The interaction between one or more biological units and one ormore surfaces can be assessed using a magnifying device and a suitablesensor to acquire images of the interaction over time. Images can becharacterized using approaches to “whole characterization” such as selfsimilarity. Images can also be characterized by identifying the objectsin the image by virtue of motion vector colocomotion, and subsequentcharacterization of each object's adherence, morphological change,motility, cell division, or cell death, as described above.

[0265] In a related embodiment, biological units are exposed to one ormore treatments while they are interacting with one or more surfaces,and those biological units are subsequently evaluated for theirpropensity to interact with the structure. An example of such a processis a the exposure of platelets to a monoclonal antibody while they areinteracting with a glass surface coated with collagen. The assessment ofthe effect of a treatment(s) on the interaction between the biologicalunit and the surface is performed using a magnifying device and asuitable sensor to acquire images of the interaction over time. Imagescan be characterized using approaches to “whole characterization” suchas self similarity. Images can also be characterized by identifying themoving objects in the image by virtue of motion vector colocomotion, andsubsequent characterization of each object's adherence, morphologicalchange, motility, cell division, or cell death, as described above.

[0266] In another embodiment, the methods and/or apparatus as describedherein can be used to evaluate the propensity of one or more biologicalunits to infiltrate a structure such as a prosthetic device. Examples ofsuch prosthetic devices include, but are not limited to, false teeth,artificial jaw implants, artificial limbs and eyes, porcine and humancardiac valves, mastectomy implants, cochlear implants, orthopedichardware, e.g. artificial joints. Such structures can be fabricated fromone or more substances that could include, but are not limited to,stainless steel, titanium, ceramic, and synthetic polymers. Theinfiltration of the prosthetic device by the biological unit is assessedusing a magnifying device and a suitable sensor to acquire images of theinteraction over time. Images can be characterized using approaches to“whole characterization” such as self similarity. Images can also becharacterized by identifying the moving objects in the image by virtueof motion vector colocomotion, and subsequent characterization of eachobject's adherence, morphological change, motility, cell division, orcell death, as described above.

[0267] In a related embodiment, biological units are exposed to one ormore treatments while they are interacting with a prosthetic device, andthose biological units are subsequently evaluated for their propensityto interact with the structure. An example of such a process is a whiteblood cell infiltrating a porcine valve in response to a chemokinenormally produced by inflammation at the site of implantation. Theassessment of the effect of a treatment(s) on the infiltration of theprosthetic device by the biological unit is performed using a magnifyingdevice and a suitable sensor to acquire images of the interaction overtime. Images can be characterized using approaches to “wholecharacterization” such as self similarity. Images can also becharacterized by identifying the moving objects in the image by virtueof motion vector colocomotion, and subsequent characterization of eachobject's adherence, morphological change, motility, cell division, orcell death, as described above.

[0268] Databases

[0269] Images and numerical data from experiments described in theabovementioned embodiments can be stored in a database or in multipledatabases, both of which will collectively be referred to as a“database” hereafter. Numerical data can include, but is not limited to,eigenvalues, self-similarity, positional information, speed, direction,intensity, number, size. Images can include all the images from theanalysis of a dynamic biological system or a subset of the images,either selected using some predetermined rule or based on attentiveacquisition and storage. By way of non-limiting example, images andnumerical data from screens, e.g., primary, secondary orstructure-activity relationship screens, as well as experiments designedto assess gene function can be entered into one or more databases. Adatabase can also contain meta-data generated during the experiments,e.g., information on the state of each cell. Furthermore, a database canalso contain annotation, e.g., experimental conditions, manipulations ortreatments under consideration, as well as information from thepublished literature on components of the experiment, either enteredmanually or using automated methods. Information contained in such adatabase can be used to catalog information, or to provide a furtherunderstanding of each manipulation or treatment based on its behavior inmultiple different screens or experimental situations, e.g., to identifywhich manipulations and treatments cause cell division as well as cellmotility, when that is considered to be more desirable or less desirablethan just causing cell motility alone. Information contained in such adatabase can also be used to match images or numerical data from geneticor chemical modulation of known targets with results derived fromscreens of uncharacterized manipulations or treatments. In this way,such a database can be used to identify the unknown target(s) ofmanipulations or treatments based on an attribute(s) shared with imagesor numerical data from the modulation of known targets.

[0270] The database can be any kind of storage system capable of storingvarious data for each of the records as described herein. In preferredembodiments, the database is a computer medium having a plurality ofdigitally encoded data records. The data record can be structured as atable, e.g., a table that is part of a database such as a relationaldatabase (e.g., a SQL database of the Oracle or Sybase databaseenvironments).

[0271] As used herein, “machine-readable media” refers to any mediumthat can be read and accessed directly by a machine, e.g., a digitalcomputer or analogue computer. Non-limiting examples of a computerinclude a desktop PC, laptop, mainframe, server (e.g., a web server,network server, or server farm), handheld digital assistant, pager,mobile telephone, and the like. The computer can be stand-alone orconnected to a communications network, e.g., a local area network (suchas a VPN or intranet), a wide area network (e.g., an Extranet or theInternet), or a telephone network (e.g., a wireless, DSL, or ISDNnetwork). Machine-readable media include, but are not limited to:magnetic storage media, such as floppy discs, hard disc storage medium,and magnetic tape; optical storage media such as CD-ROM; electricalstorage media such as RAM, ROM, EPROM, EEPROM, flash memory, and thelike; and hybrids of these categories such as magnetic/optical storagemedia.

[0272] A variety of data storage structures are available to a skilledartisan for creating a machine-readable medium having recorded thereonthe data described herein. The choice of the data storage structure willgenerally be based on the means chosen to access the stored information.In addition, a variety of data processor programs and formats can beused to store the information of the present invention on computerreadable medium.

[0273] It is to be understood that while the invention has beendescribed in conjunction with the detailed description thereof, theforegoing description is intended to illustrate and not limit the scopeof the invention, which is defined by the scope of the appended claims.Other aspects, advantages, and modifications are within the scope of thefollowing claims.

What is claimed is:
 1. A method of evaluating a dynamic system,comprising a. acquiring a plurality of images representative of thedynamic system in two or more dimensions; b. determining self-similarityamong a representative set of images; and c. characterizing the set ofimages as a statistical function of self-similarity; thereby evaluatingthe dynamic system.
 2. The method of claim 1, wherein the dimensionsinclude any of time, space, frequency spectrum, temperature, presence orabsence of an attribute of the system.
 3. The method of claim 1,wherein: the determining step includes determining self-similaritybetween all of the plurality of images; and the characterizing stepincludes characterizing the dynamic system as a statistical function ofthe self-similarities determined with respect to the plurality ofimages.
 4. The method of claim 1, wherein the images are acquired by amethod comprising: a. acquiring images at a first acquisitionparameterization; b. determining similarity between a selected image andat least one of the other images; c. characterizing the images as astatistical function of self-similarity; and the acquisitionparameterization is adjusted as a function of the self-similarity of theimages.
 5. A method of evaluating a dynamic system, comprising a.acquiring a plurality of images representative of the dynamic systemover time; b. determining self-similarity among a representative set ofimages, and c. characterizing the set of images as a statisticalfunction of self-similarity; thereby evaluating the dynamic system. 6.The method of claim 5, wherein the determining step includes determiningself-similarity between all of the plurality of images; and thecharacterizing step includes characterizing the dynamic system as astatistical function of the self-similarities determined with respect tothe plurality of images.
 7. The method of claim 5, wherein the imagesare acquired by a method comprising: a. acquiring images at a firstacquisition parameterization; b. determining similarity between aselected image and at least one of the other images; c. characterizingthe images as a statistical function of self-similarity; and d. theacquisition parameterization is adjusted as a function of theself-similarity of the images.
 8. The method of claim 5, wherein thestatistical function is a measure of entropy.
 9. The method of claim 8,wherein the statistical function is Shannon's entropy function.
 10. Themethod of claim 8, wherein the statistical function is H _(j) =−ΣP _(j)log ₂(P _(j))/log2(n), where n is number of frames
 11. The method ofclaim 5, wherein the acquiring step includes acquiring an image from asensor.
 12. The method of claim 11, wherein the sensor is a video cameraor other device suitable for acquisition of spatiotemporal or othersignals, regardless of whether those signals represent the visualspectrum.
 13. The method of claim 5, wherein the determining stepincludes determining pair-wise correlations between images.
 14. Themethod of claim 13, wherein the determining step includes determiningpair-wise correlations between a plurality of images that comprise awindow of length n images.
 15. The method of claim 14, wherein thedetermining step includes approximating a correlation between imagesseparated by more than n by treatment of intervening pair-wisecorrelations as transitional probabilities.
 16. The method of claim 13,wherein the determining step includes determining long-term andshort-term pair-wise correlations between images.
 17. The method ofclaim 5, wherein the determining step includes generating a matrix ofthe similarities.
 18. The method of claim 17, wherein the determiningstep includes generating a matrix that is any of square, normalized,comprised of probabilities, and has a diagonal of ones.
 19. The methodof claim 17, wherein the matrix is a correlation matrix.
 20. The methodof claim 17, wherein the characterizing step includes applying a matrixoperation to the matrix in order to characterize the dynamic system. 21.The method of claim 5, wherein the acquiring step includes any of (i) animage captured by a sensor, and (ii) a processed form of an imagecaptured by a sensor.
 22. The method of claim 21, wherein the processedform of the image is any of (i) a filtered form of an image captured bythe sensor, (ii) a windowed form of the image captured by the sensor,(iii) a sub-sampled form of the image, (iv) an integration of imagescaptured by the sensor over time, (v) an integration of a square ofimages captured by the sensor over time, (vi) a gradient-direction formof the image, or (vii) a combination thereof.
 23. A method of acquiringimages representative of a dynamic system, the method comprising: a.acquiring, at a selected acquisition parameterization, a plurality ofimages representative of the dynamic system over time; b. determiningsimilarity between a selected image and at least one of the otherimages; c. characterizing the images as a statistical function ofself-similarity; d. adjusting the acquisition or storageparameterization as a function of the self-similarity of the images. 24.A method of claim 23, wherein the adjusting step includes setting anacquisition parameterization to drive the statistical function to apredetermined level.
 25. A method of claim 24, wherein the adjustingstep includes setting an acquisition parameterization so that at leastone or more most recently acquired images reflects a predetermined rateof change.
 26. A method of claim 25, wherein the adjusting step includessetting an acquisition parameterization so that at least one or moremost recently acquired images reflects a predetermined rate of motion,shape change, focal change, temperature change, intensity change. 27.The method of claim 23, wherein the images are acquired by a methodcomprising: a. acquiring images at a first acquisition parameterization;b. determining similarity between a selected image and at least one ofthe other images; c. characterizing the images as a statistical functionof self-similarity; and d. adjusting the acquisition or storageparameterization as a function of the self-similarity of the images. 28.The method of claim 23, wherein the acquisition parameterizationincludes any of acquisition rate, exposure, aperture, focus, binning, orother parameter.
 29. The method of claim 23, wherein the selected imageis a more recently acquired image.
 30. The method of claim 23,comprising buffering for potential processing at least selected ones ofthe acquired images.
 31. The method of claim 30, comprising processingat least selected ones of the buffered images.
 32. The method of claim23, comprising storing at least selected ones of the acquired images.33. A method of determining movement of an object, comprising a.acquiring a plurality of images of the object; b. selecting a window ofinterest in a selected image, the selecting step including performing atleast one autocorrelation between a candidate window and a region inwhich the candidate window resides in the selected image; c. identifyingmovement of the object as function of a cross-correlation between thewindow of interest and corresponding window in another of the images.34. The method of claim 33, wherein the images are acquired by a methodcomprising: a. acquiring images at a first acquisition parameterization;b. determining similarity between a selected image and at least one ofthe other images; c. characterizing the images as a statistical functionof self-similarity; and d. the acquisition parameterization is adjustedas a function of the self-similarity of the images.
 35. The method ofclaim 33, wherein the identifying step includes performing at least oneautocorrelation between a candidate corresponding window in anotherimage and a region in that image in which that candidate window resides.36. The method of claim 33, wherein the identifying step includesfinding maxima in the cross-correlation.
 37. A method of determiningmovement of an object, comprising a. acquiring a plurality of images ofthe object; b. selecting a window of interest in a selected image, theselecting step including performing at least one autocorrelation betweena candidate window and a region in which the candidate window resides inthe selected image; c. estimating at least one autocorrelation on awindow that corresponds to the window of interest in another of theimages; d. identifying movement of the object as function ofdisplacement of the characterizing portions of the autocorrelations. 38.The method of claim 37, wherein the images are acquired by a methodcomprising: a. acquiring images at a first acquisition parameterization;b. determining similarity between a selected image and at least one ofthe other images; c. characterizing the images as a statistical functionof self-similarity; and d. the acquisition parameterization is adjustedas a function of the self-similarity of the images.
 39. The method ofclaim 37, wherein the identifying step further includes matching atleast characterizing portions of the autocorrelations.
 40. A method ofanalyzing motion in a plurality of images, comprising a. acquiring aplurality of images, b. selecting a plurality of windows of interest ina selected image, the selecting step including estimating, for eachwindow of interest, at least one autocorrelation between a candidatewindow and a region in which the candidate window resides in theselected image; c. identifying motion vectors as function of across-correlation between each window of interest and a correspondingwindow in another of the images.
 41. The method of claim 40, wherein theimages are acquired by a method comprising: a. acquiring images at afirst acquisition parameterization; b. determining similarity between aselected image and at least one of the other images; c. characterizingthe images as a statistical function of self-similarity; and d. theacquisition parameterization is adjusted as a function of theself-similarity of the images.
 42. The method of claim 40, wherein theidentifying step includes estimatinging at least one autocorrelationbetween a candidate corresponding window in another image and a regionin that image in which that candidate window resides.
 43. The method ofclaim 40, wherein the identifying step includes finding maxima in thecross-correlations.
 44. The method of claim 40, further comprisingsegmenting the image as a function of the motion vectors.
 45. The methodof claim 44, wherein the segmenting step includes finding one or moresets of motion vectors with minimum square distances with respect to oneanother.
 46. A method of analyzing motion in a plurality of images, themethod comprising a. acquiring a plurality of images of the object; b.selecting a plurality of windows of interest in a selected image, theselecting step including estimating, for each window of interest, atleast one autocorrelation between a candidate window and a region inwhich the candiate window resides in the selected image; c. for eachwindow of interest, performing at least one autocorrelation on arespective corresponding window in another image; d. identifying motionvectors as functions of displacements of the autocorrelations of eachwindow of interest and the corresponding window in the another image.47. The method of claim 46, wherein the images are acquired by a methodcomprising: a. acquiring images at a first acquisition parameterization;b. determining similarity between a selected image and at least one ofthe other images; c. characterizing the images as a statistical functionof self-similarity; and d. the acquisition parameterization is adjustedas a function of the self-similarity of the images.
 48. The method ofclaim 46, wherein the identifying step further includes matching atleast characterizing portions of the autocorrelations.
 49. The method ofclaim 46, further comprising segmenting the image as a function of themotion vectors.
 50. The method of claim 46, wherein the segmenting stepincludes finding one or more sets of motion vectors with minimum squaredistances with respect to one another.
 51. The method of claim 1,wherein the dynamic system is a dynamic biological system comprising abiological unit.
 52. The method of claim 51, wherein the biological unitis undergoing morphological change.
 53. The method of claim 52, whereinthe morphological change is selected from the group consisting of celldifferentiation, cell motility cell spreading, cell contraction, cellphagocytosis, cell pinocytosis, cell exocytosis, cell growth, celldeath, cell division, cell polarization, organismal motility, andorganismal development.
 54. The method of claim 51, wherein thebiological unit is motile.
 55. The method of claim 53, wherein thebiological unit is undergoing cell division.
 56. The method of claim 55,wherein the biological unit is undergoing meiosis or mitosis.
 57. Themethod of claim 51, wherein the biological unit is undergoing celladherence.
 58. The method of claim 51, wherein the biological unit isadjacent to, in contact with, or adhered to a second entity during imageacquisition.
 59. The method of claim 58, wherein the second entity is asurface or another biological unit.
 60. The method of claim 51, whereinthe biological unit is selected from the group consisting of biologicalpolymers, carbohydrates, lipids, and ions.
 61. The method of claim 51,wherein the biological unit is labeled.
 62. The method of claim 61,wherein the label is selected from the group consisting of magnetic ornon-magnetic beads, antibodies, fluorophores, radioemitters, and labeledligands.
 63. The method of claim 62, wherein the radioemitter isselected from the group consisting of an alpha emitter, a beta emitter,a gamma emitter, or a beta- and gamma-emitter.
 64. The method of claim5, wherein the dynamic system is a dynamic biological system comprisinga biological unit.
 65. The method of claim 64, wherein the biologicalunit is undergoing morphological change.
 66. The method of claim 65,wherein the morphological change is selected from the group consistingof cell differentiation, cell motility cell spreading, cell contraction,cell phagocytosis, cell pinocytosis, cell exocytosis, cell growth, celldeath, cell division, cell polarization, organismal motility, andorganismal development.
 67. The method of claim 64, wherein thebiological unit is motile.
 68. The method of claim 66, wherein thebiological unit is undergoing cell division.
 69. The method of claim 68,wherein the biological unit is undergoing meiosis or mitosis.
 70. Themethod of claim 64, wherein the biological unit is undergoing celladherence.
 71. The method of claim 64, wherein the biological unit isadjacent to, in contact with, or adhered to a second entity during imageacquisition.
 72. The method of claim 71, wherein the second entity is asurface or another biological unit.
 73. The method of claim 64, whereinthe biological unit is selected from the group consisting of biologicalpolymers, carbohydrates, lipids, and ions.
 74. The method of claim 73,wherein the biological unit is labeled.
 75. The method of claim 74,wherein the label is selected from the group consisting of magnetic ornon-magnetic beads, antibodies, fluorophores, radioemitters, and labeledligands.
 76. The method of claim 75, wherein the radioemitter isselected from the group consisting of an alpha emitter, a beta emitter,a gamma emitter, or a beta- and gamma-emitter.
 77. The method of claim23, wherein the dynamic system is a dynamic biological system comprisinga biological unit.
 78. The method of claim 77, wherein the biologicalunit is undergoing morphological change.
 79. The method of claim 78,wherein the morphological change is selected from the group consistingof cell differentiation, cell motility cell spreading, cell contraction,cell phagocytosis, cell pinocytosis, cell exocytosis, cell growth, celldeath, cell division, cell polarization, organismal motility, andorganismal development.
 80. The method of claim 77, wherein thebiological unit is motile.
 81. The method of claim 79, wherein thebiological unit is undergoing cell division.
 82. The method of claim 81,wherein the biological unit is undergoing meiosis or mitosis.
 83. Themethod of claim 77, wherein the biological unit is undergoing celladherence.
 84. The method of claim 77, wherein the biological unit isadjacent to, in contact with, or adhered to a second entity during imageacquisition.
 85. The method of claim 84, wherein the second entity is asurface or another biological unit.
 86. The method of claim 77, whereinthe biological unit is selected from the group consisting of biologicalpolymers, carbohydrates, lipids, and ions.
 87. The method of claim 77,wherein the biological unit is labeled.
 88. The method of claim 87,wherein the label is selected from the group consisting of magnetic ornon-magnetic beads, antibodies, fluorophores, radioemitters, and labeledligands.
 89. The method of claim 88, wherein the radioemitter isselected from the group consisting of an alpha emitter, a beta emitter,a gamma emitter, or a beta- and gamma-emitter.
 90. A method ofevaluating an attribute of a biological unit over time, the methodcomprising: a. providing a plurality of images representative of thebiological unit over time; b. evaluating the similarity between aselected image and one of the other images to determine a pairwisesimilarity measurement; c. generating a self-similarity matrixcomprising the pairwise similarity measurement; and d. characterizingthe biological unit as a function of the self-similarity matrix, therebyevaluating the attribute of the biological system.
 91. The method ofclaim 90, wherein the images are acquired by a method comprising: a.acquiring images at a first acquisition parameterization; b. determiningsimilarity between a selected image and at least one of the otherimages; c. characterizing the images as a statistical function ofself-similarity; and the acquisition parameterization is adjusted as afunction of the self-similarity of the images.
 92. The method of claim90, wherein the attribute is selected from the group consisting of cellmorphology, migration, motility, death, binding to or interacting with asecond entity, and division.
 93. The method of claim 90, wherein thedetermining comprises determining similarity between the selected imageand all of the other images.
 94. The method of claim 90, furthercomprising selecting a plurality of images and evaluating the similaritybetween pairs of images to determine a pairwise similarity measurement,and generating a self-similarity matrix comprising the pairwisesimilarity measurements.
 95. The method of claim 90, further comprisingselecting a plurality of the images and evaluating the similaritybetween all the images to determine a pairwise similarity measurement,and generating a self-similarity matrix comprising the pairwisesimilarity measurements.
 96. The method of claim 90, wherein thecharacterizing step comprises generating eigenvalues from theself-similarity matrix.
 97. The method of claim 90, wherein thecharacterizing step comprises generating entropic indices from theself-similarity matrix.
 98. A method of evaluating an attribute of adynamic biological system over time, the method comprising: a. providinga plurality of images representative of the dynamic biological system;b. generating a motion field from at least two images; and c.characterizing the dynamic biological system as a statistical functionof the motion field, thereby evaluating the dynamic biological system.99. The method of claim 98, wherein the characterizing step comprises astatistical analysis of motion vectors.
 100. The method of claim 99,wherein the characterizing step comprises evaluating direction orvelocity in the dynamic biological system.
 101. The method of claim 100,wherein the statistical analysis comprises evaluating direction andvelocity in the dynamic biological system.
 102. The method of claim 101,further comprising determining the distribution of direction or velocityin the dynamic biological system.
 103. The method of claim 101, furthercomprising statistical analysis of velocity as a function of direction.104. The method of claim 101, further comprising statistical analysis ofdirection as a function of velocity.
 105. The method of claim 98,wherein the characterization further comprises detecting one or moremoving objects in the image.
 106. The method of claim 105, wherein theobjects are detected based on motion vector colocomotion.
 107. Themethod of claim 105, further comprising determining the direction orvelocity of the moving object as a function of colocomoting motionvectors.
 108. The method of claim 105, further comprising determiningthe direction and velocity of the moving object as a function ofcolocomoting motion vectors.
 109. The method of claim 108, furthercomprising statistical analysis of velocity as a function of direction.110. The method of claim 108, further comprising statistical analysis ofdirection as a function of velocity.
 111. The method of claim 105,further comprising determining the center of motion for a moving object.112. The method of claim 111, further comprising determining thedirectional persistence of the moving object.
 113. The method of claim111, further comprising determining the direction or velocity of thecenter of motion of the moving object.
 114. The method of claim 111,further comprising determining the direction and velocity of the centerof motion of the moving object.
 115. The method of claim 113, furthercomprising statistical analysis of velocity as a function of direction.116. The method of claim 113, further comprising statistical analysis ofdirection as a function of velocity.
 117. The method of claim 113,further comprising determining the distribution of direction or velocityof a moving object.
 118. The method of claim 105, further comprisingestablishing a bounding box for a moving object.
 119. The method ofclaim 118, further comprising establishing a bounding box for eachmoving object.
 120. The method of claim 118, wherein the bounding boxcorresponds exactly to the maximum dimensions of the object.
 121. Themethod of claim 118, wherein the bounding box corresponds to the maximumdimensions of the object plus a preselected factor.
 122. The method ofclaim 118, wherein the size of the bounding box varies with theself-similarity of the object.
 123. The method of claim 118, furthercomprising analyzing the area within the bounding box.
 124. The methodof claim 123, wherein the analyzing step is selected from a groupconsisting of applying image segmentation based on raw intensity,texture, and frequency.
 125. The method of claim 105, further comprisingevaluating an attribute of the object.
 126. The method of claim 125,wherein the evaluating comprises: a. providing a plurality of images ofthe object; b. evaluating the similarity between a plurality of imagesof the object; and c. characterizing the object as a function of thesimilarity between the images.
 127. The method of claim 126, wherein theplurality of images is a pair of images.
 128. The method of claim 126,wherein the characterizing comprises generating a self-similaritymatrix.
 129. The method of claim 126, wherein the plurality of images ofthe object comprises images of the area within the bounding box. 130.The method of claim 105, further comprising calculating the dimensionsof the object.
 131. The method of claim 130, wherein the dimensions ofthe object are a major axis and a minor axis.
 132. The method of claim131, further comprising characterizing the shape of the object as afunction of the major axis and the minor axis.
 133. The method of claim128, further comprising generating eigenvalues.
 134. A method ofcharacterizing a dynamic biological system comprising a biological unit,the method comprising: a. providing the dynamic biological system; b.acquiring a plurality of images representative of the dynamic biologicalsystem in two dimensions; c. determining self-similarity between arepresentative set of the images; and d. characterizing the set ofimages as a statistical function of self-similarity, therebycharacterizing the dynamic biological system.
 135. The method of claim134, wherein the plurality of images are acquired by a methodcomprising: a. acquiring the images at a first acquisitionparameterization; b. determining self-similarity between a selectedimage and at least one of the other images; c. characterizing the imagesas a statistical function of self-similarity; and d. adjusting theacquisition or storage parameterization as a function of theself-similarity of the images.
 136. The method of claim 134, wherein thedynamic biological system comprises a plurality of biological units.137. The method of claim 135, wherein the biological units areindependently selected from one or more of cells, tissue, organs, andunicellular organisms, multicellular organisms.
 138. The method of claim134, wherein the characterizing provides information regarding one ormore attributes of the biological unit.
 139. The method of claim 134,wherein the biological unit is a cell.
 140. The method of claim 139,wherein the one or more attributes are selected from the groupconsisting of cell motility, cell morphology, cell division, celladherence.
 141. The method of claim 134, wherein the biological unit isan organism.
 142. The method of claim 141, wherein the one or moreattributes are selected from the group consisting of organismalmotility, organismal morphological change, organismal reproduction, andthe movement or morphological change of individual tissues or organswithin an organism.
 143. The method of claim 134, wherein the dynamicbiological system is manipulated.
 144. The method of claim 143,comprising acquiring a plurality of images representative of the dynamicbiological system at one or more of the following points: prior to,concurrently with, and subsequent to the manipulation.
 145. The methodof claim 143, wherein the manipulation is selected from the groupconsisting of alterations in temperature, viscosity, shear stress, celldensity, oxygen tension, carbon dioxide tension, composition of media orsurfaces contacted, electrical charge, gene expression, proteinexpression, addition of one or more other biological units of the sameor different type, and the addition or removal or one or moretreatments.
 146. The method of claim 143, wherein the manipulation isthe addition or removal of a treatment.
 147. The method of claim 146,wherein the treatment is exposure of the dynamic biological system atest compound.
 148. The method of claim 147, wherein the test compoundis selected from the group consisting of small molecule, nucleic acids,proteins, antibodies, sugars and lipids.
 149. The method of claim 143,wherein a plurality of dynamic biological systems is each exposed to adifferent manipulation.
 150. The method of claim 149, wherein aredundant set of dynamic biological systems is exposed to a redundantset of manipulations.
 151. The method of claim 143, further comprisingevaluating the effect of the manipulation on one or more attributes ofthe one or more biological units.
 152. The method of claim 146, furthercomprising evaluating the effect of the treatment on one or moreattributes of the one or more biological units.
 153. The method of claim151, wherein the one or more biological units comprise one or morecells, and the one or more attributes are selected from the groupconsisting of cell motility, cell morphological change, cell adherence,cell division.
 154. The method of claim 151, wherein the one or morebiological units comprise one or more organisms and the one or moreattributes are selected from the group consisting of organismalmotility, organismal morphological change, organismal reproduction, andthe movement or morphological change of individual tissues or organswithin an organism.
 155. The method of claim 134, wherein the dynamicbiological system comprises a plurality of biological units that are allsimilar.
 156. The method of claim 134, wherein the dynamic biologicalsystem comprises two or more different biological units.
 157. The methodof claim 151, further comprising evaluating the effect of themanipulation on an attribute of a biological unit, and selecting themanipulation for further analysis.
 158. The method of claim 157, whereinthe further analysis is by the method of claim
 1. 159. The method ofclaim 157, wherein the further analysis is by a method other than amethod of evaluating a dynamic biological system comprising providingthe biological unit; acquiring a plurality of images representative ofthe dynamic system in two dimensions; determining self-similaritybetween a representative set of images; and characterizing the images asa statistical function of self-similarity.
 160. The method of claim 159,wherein the manipulation is the addition or removal of a treatment. 161.The method of claim 159, wherein the further analysis is by a highthroughput or parallel screen.
 162. The method of claim 161, wherein thescreen comprises evaluating a test compound for its ability to interactwith a receptor or other target.
 163. The method of claim 162, whereinthe screen is selected from the group consisting of combinatorialchemistry, computer-based structural modeling and rational drug design.164. The method of claim 162, wherein the screen is selected from thegroup consisting of determining the binding affinity of the testcompound, phage display, cell display, and drug western.
 165. The methodof claim 143, wherein the manipulation was identified in prior screen.166. The method of claim 165, wherein the prior screen was by a methodwhich does not acquire a plurality of images representing a dynamicsystem in two dimensions.
 167. A method of optimizing the effect of atest compound on an attribute of a biological unit, the methodcomprising: a. selecting a first test compound; b. exposing a dynamicbiological system to the first test compound; c. acquiring a pluralityof images representative of the dynamic biological system in twodimensions; d. determining self-similarity between a representative setof the images; e. characterizing the set of images as a statisticalfunction of self-similarity; f. providing a next generation testcompound; g. exposing a dynamic biological system to the next generationtest compound; h. acquiring a plurality of images representative of thedynamic biological system in two dimensions; i. determining similaritybetween a representative set of the images; and j. characterizing theset of images as a statistical function of self-similarity, and k.repeating steps f-j with successive next generation compounds, therebyoptimizing the effect of a test compound on an attribute.
 168. Themethod of claim 167, wherein one or more of the first test compound andthe next generation compound are selected from a database of compoundsof known chemical structure.
 169. The method of claim 167, wherein thenext generation compound is a variant of the first test compound. 170.The method of claim 152, further comprising:
 1. selecting a firsttreatment;
 2. providing a next generation treatment;
 3. exposing adynamic biological system to the next generation treatment;
 4. acquiringa plurality of images representative of the dynamic biological system intwo dimensions;
 5. determining self-similarity between a representativeset of the images; and
 6. characterizing the plurality of images as astatistical function of self-similarity.
 171. The method of claim 170,comprising acquiring a plurality of images representative of the dynamicbiological system at one or more of the following points: prior to,concurrently with, and subsequent to the exposure to the next generationtreatment.
 172. The method of claim 156, wherein the biological unitsdiffer genetically, epigenetically, phenotypically or in developmentalstage.
 173. The method of claim 156, wherein the biological units differas a result of manipulation.
 174. The method of claim 173, wherein themanipulation is a treatment.
 175. The method of claim 174, wherein thetreatment is exposure to a test compound.
 176. The method of claim 172,wherein the genetic difference comprises gene deletion or duplication,targeted mutation, random mutation, introduction of additional geneticmaterial.
 177. A method of determining the relationship between aproperty of a treatment, or a series of treatments, and the ability tomodulate an attribute of a biological unit, the method comprising: a.providing a first test compound having a first property; b. exposing adynamic biological system comprising a biological unit to the first testcompound; c. acquiring a plurality of images representative of thedynamic biological system in two dimensions; d. determiningself-similarity between a representative set of the images; e.characterizing the set of images as a statistical function ofself-similarity; f. providing a second test compound having at least oneproperty similar to a property of the first treatment and at least oneproperty that differs; g. exposing a dynamic biological systemcomprising a biological unit to the second test compound; h. acquiring aplurality of images representative of the dynamic biological system intwo dimensions; i. determining self-similarity between a representativeset of the images; j. characterizing the set of images as a statisticalfunction of self-similarity; and k. correlating the similar property ofthe first and second test compounds with an effect on one or moreattributes.
 178. The method of claim 177, wherein the property isselected from the group consisting of chemical structure, nucleic acidsequence, amino acid sequence, phosphorylation, methylation, sulfation,nitrosylation, oxidation, reduction, affinity, carbohydrate structure,lipid structure, charge, size, bulk, isomerization; enantiomerization;and rotational property of a selected moiety.
 179. A method ofevaluating or selecting a target, the method comprising: a. providing afirst test compound; b. contacting a dynamic biological systemcomprising a biological unit with the first test compound; and c.performing a method comprising:
 1. acquiring a plurality of imagesrepresentative of the dynamic biological system in two dimensions; 2.determining self-similarity between a representative set of the images;and
 3. characterizing the set of images as a statistical function ofself-similarity; thereby providing a value for a parameter related tothe effect of the first test compound on the selected attribute; d.providing a second test compound; e. contacting one or more biologicalunits with the second test compound; f. performing a method comprising:4. acquiring a plurality of images representative of the dynamicbiological system in two dimensions;
 5. determining self-similaritybetween a representative set of the images; and
 6. characterizing theset of images as a statistical function of self-similarity; therebyproviding a value for a parameter related to the effect of the secondtest compound on the selected attribute; and g. comparing the parametersand selecting the test compound having the desired effect on theattribute, thereby selecting a target.
 180. A method of evaluating theactivity of a gene, the method comprising: a. providing a firstreference biological unit or plurality thereof; b. providing a secondbiological unit or plurality thereof wherein the activity of the gene ismodulated as compared to the first biological unit, and c. performing amethod comprising:
 1. acquiring a plurality of images representative ofthe dynamic biological system in two dimensions;
 2. determiningself-similarity between a representative set of the images; and 3.characterizing the set of images as a statistical function ofself-similarity; thereby evaluating the activity of the gene.
 181. Themethod of claim 180, wherein the gene is modulated by directed or randommutagenesis.
 182. The method of claim 180, wherein a plurality of genesare modulated.
 183. The method of claim 182, wherein the plurality ofgenes are modulated by random mutagenesis.
 184. The method of claim 180,wherein the plurality of genes are selected from the results of anexpression profile experiment.
 185. The method of claim 180, wherein aplurality of genes are modulated in a plurality of biologicalunits/dynamic systems.
 186. The method of claim 182, wherein a uniquegene is modulated in each of a plurality of biological units/dynamicsystems.
 187. The method of claim 180, further comprising manipulatingthe dynamic system and evaluating the effect of the manipulation on theactivity of the gene.
 188. A method of evaluating the interaction of abiological unit with a surface, the method comprising: a. providing adynamic biological system comprising a biological unit; b. contactingthe dynamic biological system with a surface; and c. performing a methodcomprising:
 1. acquiring a plurality of images representative of thedynamic biological system in two dimensions;
 2. determiningself-similarity between a representative set of the images; and 3.characterizing the set of images as a statistical function ofself-similarity; thereby evaluating the interaction of the biologicalunit with the surface.
 189. The method of claim 188, wherein the surfaceis uniform.
 190. The method of claim 188, wherein the surface isvariable.
 191. The method of claim 188, wherein the surface comprisespores, openings, concavities, convexities, smooth areas, and roughareas, etchings, lithographed patterns.
 192. The method of claim 188,wherein the surface variability comprises changes in composition. 193.The method of claim 188, wherein the surface variability compriseschanges in charge.
 194. The method of claim 188, wherein the interactionis selected from the group consisting of adherence to the surface,movement across the surface, release from the surface; deposit orremoval of a material on the surface, and infiltration of pores oropenings.
 195. The method of claim 188, wherein the surface variabilitycomprises the presence or absence of a test compound.
 196. The method ofclaim 195, wherein the test compound is present in a gradient.
 197. Amethod for evaluating the propensity of one or more biological units tointeract with, the method comprising: a. providing one or morebiological units; b. providing a structure; c. performing a methodcomprising:
 1. acquiring a plurality of images representative of thedynamic biological system in two dimensions;
 2. determiningself-similarity between a representative set of the images; and 3.characterizing the set of images as a statistical function ofself-similarity, thereby evaluating the propensity of the biologicalunits to infiltrate a structure.
 198. The method of claim 197, whereinthe structure is a prosthetic device.
 199. The method of claim 197,wherein the structure is selected from the group consisting of stainlesssteel, titanium, ceramic, and synthetic polymer.
 200. The method ofclaim 197, further comprising exposing the biological units to a testcompound and evaluating the effect of the test compound on thepropensity of the biological units to infiltrate the structure.
 201. Amethod of evaluating the interaction between a biological unit and asecond entity, the method comprising: a. providing one or morebiological units; b. providing a second entity; c. performing a methodcomprising:
 1. acquiring a plurality of images representative of thedynamic biological system in two dimensions;
 2. determiningself-similarity between a representative set of the images; and 3.characterizing the set of images as a statistical function ofself-similarity, thereby evaluating the interaction of the biologicalunits and the second entity.
 202. The method of claim 134, wherein thebiological unit is in a single well.
 203. The method of claim 134,wherein the plurality of images representative of the dynamic system areimages of a single biological unit.
 204. A method of evaluating a testcompound, the method comprising: a. providing a first biological unit orplurality thereof; b. providing a second biological unit or pluralitythereof, that is the same as the first biological unit or pluralitythereof; c. contacting the second biological agent with the testcompound;
 1. performing a method comprising acquiring a plurality ofimages representative of the dynamic biological system in twodimensions;
 2. determining self-similarity between a representative setof the images; and
 3. characterizing the set of images as a statisticalfunction of self-similarity; and d. comparing the attributes of thebiological unit in the presence and absence of the test compound,thereby evaluating the test compound.
 205. The method of claim 204,further comprising: a. providing a second test compound; b. contactingthe first biological unit with the second test compound; c. performing amethod comprising:
 1. acquiring a plurality of images representative ofthe dynamic biological system in two dimensions;
 2. determiningself-similarity between a representative set of the images; and 3.characterizing the set of images as a statistical function ofself-similarity; and d. comparing the attributes of the biological unitin the presence and absence of the test compound.
 206. The method ofclaim 134 wherein the biological units are on an addressable array. 207.An apparatus comprising: a. a sensor configured to acquire imagesrepresentative of a dynamic system at an adjustable parameterization; b.a storage device configured to store the images at an adjustableparameterization; and c. a data processing device configured to analyzesimilarities between the images.
 208. The apparatus of claim 207,further comprising a display device.
 209. The apparatus of claim 207,wherein the data processing device is further configured to adjust theacquisition parameterization of the sensor as a statistical function ofthe similarity between images.
 210. The apparatus of claim 207, whereinthe data processing device is further configured to adjust the storageparameterization of the storage device as a statistical function of thesimilarity between images.
 211. The apparatus of claim 207, where thedata processing device is further configured to adjust the acquisitionparameterization of the sensor and the storage parameterization of thestorage device as a statistical function of the similarity betweenimages.
 212. The apparatus of claim 209 wherein the adjusting includessetting the acquisition parameterization to drive the statisticalfunction to a predetermined level.
 213. The apparatus of claim 212,wherein the adjusting step includes setting the acquisitionparameterization so that at least one or more most recently acquiredimages reflects a predetermined rate of change.
 214. The apparatus ofclaim 213, wherein the adjusting step includes setting the acquisitionparameterization so that at least one or more most recently acquiredimages reflects a predetermined rate of motion, shape change, focalchange, temperature change, or intensity change.
 215. The apparatus ofclaim 214, wherein the acquisition parameterization comprisesacquisition rate, exposure, aperture, focus, binning, or otherparameter.
 216. The apparatus of claim 207, further comprising bufferingmeans for potential processing of one or more images.
 217. The apparatusof claim 210, wherein the storage parameterization comprises imagelabeling.
 218. The apparatus of claim 210 wherein the adjusting stepincludes setting the storage parameterization so that at least one ormore recently acquired images reflects a predetermined rate of change.219. The apparatus of claim 218, wherein the adjusting step includingsetting the storage parameterization so that at least one or more mostrecently acquired images reflects a predetermined rate of motion, shapechange, focal change, temperature change, or intensity change.
 220. Theapparatus of claim 207, wherein a magnifying device is placed betweenthe scene and the sensor.
 221. The apparatus of claim 207, wherein afiltering device is placed between the scene and the sensor
 222. Themethod of claim 13, wherein the pair-wise correlations are performed bya sensor that employs Fourier optics.
 223. The method of claim 14,wherein the pair-wise correlations are performed by a sensor thatemploys Fourier optics
 224. The method of claim 16, wherein theshort-term pair-wise correlations are performed by a sensor that employsFourier optics.
 225. A database which comprises a plurality of recordswherein each record includes at least one of the following: a. data onthe identity of a biological unit; b. data on an attribute of thebiological unit; and c. data on a the effect of one or more manipulationon the attribute.
 226. The database of claim 225, wherein saidmanipulation is a treatment.
 227. The database of claim 225, wherein thetreatment is the administration of a test compound.
 228. The database ofclaim 225, wherein the data on the identity of the biological unitincludes genotypic and phenotypic information.
 229. The database ofclaim 225, wherein the genotypic information includes informationregarding the presence, absence, spatial location, or temporalexpression of a gene.
 230. The database of claim 225, wherein thegenotypic information includes information regarding the presence orabsence of one or more mutations.
 231. The database of claim 225,wherein the phenotypic data includes one or more of cell type, organismtype, cell status, age,
 232. The database of claim 225, wherein thedatabase includes at least two records, and the manipulation in each ofthe records differs from the other record.
 233. The database of claim225, wherein the manipulation is administration of a test compound andin one record the preselected factor includes administration of the testcompound and in the other record the test compound is not administeredor is administered at a different dose.
 234. The database of claim 225,wherein the database includes at least two records, and at least onemanipulation in each of the records differs from the other record. 235.The database of claim 225, wherein at least one manipulation in therecords differs and at least one of the other manipulations is the same.236. A method for identifying an unknown target, the method comprising:a. providing the database of claim 225, comprising:
 1. at least a firstrecord having data about the effect of a first manipulation on aattribute, where the target of the first test compound is known; and 2.at least a second record having data about the effect of a secondmanipulation on an attribute, where the target of the secondmanipulation is unknown; and b. comparing the data in the first recordto the data of the second record.
 237. The database of claim 225,wherein the database is in computer readable form.
 238. The method ofclaim 134, further comprising: a. selecting a plurality of windows ofinterest in a selected image, the selecting step including estimating,for each window of interest, at least one autocorrelation between acandidate window and a region in which the candidate window resides inthe selected image; and b. identifying motion vectors as function of across-correlation between each window of interest and a correspondingwindow in another of the images.
 239. The method of claim 238, furthercomprising segmenting the image as a function of the motion vectors.240. The method of claim 239, wherein the segmenting step includesfinding one or more sets of motion vectors with minimum square distanceswith respect to one another.